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Abstract 


The LP recourse problem applies to two-stage optimization problems 
where uncertainty in resource availability of the second stage hinders informed decision 
making. The recourse function affords a way to compensate “later” for an error in 
prediction “now.” The literature provides a rich body of work on the optimization of 
such problems, but little research has been accomplished regarding the characterization of 
the surface in the local region of optimality, in particular sensitivity analysis. A decision 
maker faced with considerations other than the modeled objective function must be 
presented with a way to estimate the impact of operating at non-optimal decision variable 
values. This work develops and demonstrates a technique for characterizing the surface 
using response surface methodology. Specifically, the flexibility and utility of RSM 
techniques applied to this class of problems is demonstrated, and a methodology for 
characterizing the surface in the local region using a low-order polynomial is developed. 
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1.0: Introduction 


This research extends the work of Lt. Col. T. Glenn Bailey’s Doctoral 
Dissertation at the University of Texas, Austin (Bailey 1995). Bailey utilized response 
surface methodology (RSM) in characterizing the surface of several two-stage stochastic 
linear programming with recourse (2SSLP) problems as a post-optimization analysis 
process. The problems he examined included APL1P, CEP-1, PGP2,4-Term, and 20- 
Term. With the exception of 20-Term, the problems were small (15 decision variables or 
fewer), and the RSM analysis was predicated with knowledge gained during the 
optimization process. 

The goals of this work were twofold: First, to develop a structured methodology 
for the RSM-based analysis of this class of problems independent of any optimization 
knowledge, and second, to demonstrate the methodology on “large” problems (60 or 
more decision variables). The sponsor for the research was Lt. Col. Bailey, and the 
computer resources used were the property of the Air Force Institute of Technology. 

This work develops and demonstrates a structured methodology for designing and 
analyzing a low-order polynomial approximation for the 2SSLP family of problems. 
These problems are often difficult to analyze due to high dimensionality and a highly 
variant stochastic nature. A polynomial metamodel is useful for providing a decision 
maker with sensitivity analysis, and provides an easily computed estimate for changes in 
decision variable values. 

Chapter 2 introduces the problem with a rigorous mathematical definition, and 
provides a historical background. Chapter 3 develops the methodology for the work, and 
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Chapter 4 provides computational results for two problems considered. Chapter 5 
provides an overview of results, and directions for further research. 
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2.0: Two-Stage Stochastic Linear Programming with Recourse: 

A Characterization of Local Regions using Response Surface Methodology 

This thesis develops and implements a technique for characterizing the response 

surface of a two-stage stochastic linear programming (LP) with recourse problem using 

response surface methodology (RSM). Specifically, we demonstrate the flexibility and 

utility of RSM techniques applied to this class of problems, and develop a methodology 

for characterizing the surface in a local region using a low-order polynomial. 

The LP recourse problem applies to two-stage optimization problems where 

uncertainty in resource availability of the second stage hinders informed decision making. 

The recourse function affords a way to compensate “later” for an error in prediction 

“now” (Kail and Wallace, 1994). The general form of the problem is 

Z(x) = Min[ c - x + ^(- x ’®’ T )H 0) 

JC 

where 

h(x,co,T) = Min dy (2) 

y 

3 Wv -(0 + T.x,_y > 0 

The second-stage RHS vector co and matrix T can contain both stochastic and 
deterministic elements, and W is the matrix of second-stage technological coefficients 
(Bailey, Jensen and Morton 1996). The objective is to find x*, where Z(x*) = min Z(x). 

Dantzig first proposed the recourse problem as a way to extend linear 
programming beyond deterministic here-and-now decision problems, and generalized it 
to an m-stage problem (Dantzig 1955). Dantzig notes that only the first stage activities 
are determined, the later stage variables are random and dependent on the first stage 
variables, and that for any feasible choice of first-stage variables the following stages are 
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feasible. This last property is relatively complete recourse (Kail and Wallace 1994), an 
attribute we assume for two-stage problems. Wets (1966) proved that the feasible 
probability space of the problem is both convex and continuous, thus guaranteeing a 
unique optimal solution. 

The literature provides significant research in optimization of problems in this 
class. Ermoliev (1983,1988) explored gradient and quasi-gradient optimization methods, 
and introduced the use of the dual variables for gradient information. Higle and Sen 
(1991) employed decomposition methods for optimization, and Sen, Doverspike and 
Cosares (1994) used decomposition methods and validated the optimization with a 
simulation of the system studied. Bailey (1995) used the projective gradient among other 
nonlinear optimization techniques, and studied estimator variance reduction techniques. 
Morton and Wood (1997) demonstrated techniques for bounding the optimal solution of 
the problem. 

RSM integrates statistical experiment design, regression techniques, and 
elementary optimization methods to identify and fit an appropriate response surface 
which characterizes a system that is generally not fully understood. RSM seeks to 
identify the input variables which affect an output (or response) variable, and fit an 
appropriate response surface. The response surface used is generally a first- or second- 
order polynomial fitted using the least-squares approach (see Myers and Montgomery 
1995, Box and Draper 1987). 

Bailey (1995) utilized elements of RSM on the recourse problem by showing that 
designed experiments allow a systematic low-order polynomial characterization of the 
surface. He included sampling techniques to reduce the variance of the estimator of 
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E[h(x, 0 ), T)], since excessive estimator variance often creates an indeterminate 
approximation of the known piecewise-convex surface. Bailey found the greatest 
variance reduction using Latin hypercube sampling (LHS), a method we employ in this 
paper (for the theoretical background of LHS, see Mckay et al 1979, and Mckay 1988). 
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3.0: Design Methodology 

Our approach begins with the assumption that x* is known, then continues with 
providing an appropriate experimental design to characterize the response surface in the 
optimal region. We empirically demonstrate the utility of this technique using the 
sequential design strategy depicted in Figure 3-1. 


1) Screen decision variables 

a) Design and fit Resolution IV design augmented with axial points 

b) Fit model of linear and pure quadratic terms 

c) Retain factors with predetermined significance level 

2) Choose experimental design for final model 

a) Central Composite Design 

b) Minimal Bias design 

c) Simplex Lattice if equality constraints are present 

3) Choose factor ranges and design center 

a) Design centering strategy is based on analysis objectives 

b) Factor ranges based on analysis objectives and a priori knowledge of problem 
characteristics 

4) Sample problem for experimental design, and fit model 

5) Model validation 

a) Statistical validation 

i) Residual analysis 

ii) ANOVA significance 

iii) Lack of fit 

b) Sampling validation 

i) Randomly sample design space 

ii) Compare predicted values to actual observations 

6) Canonical analysis 

a) Analyze curvature of fitted model 

b) Compare to known characteristics of problem 


Figure 3-1: General Design Methodology. 


3.1: CCD and Minimal Bias Design 

Two sources of error in an empirical model are variance and bias, the latter 
referred to as systematic errors (Myers and Montgomery 1995). Bailey (1995) addresses 
several sampling methods to reduce model variance, but left model bias for later work. 
We construct and use a minimal bias design by adapting standard minimal-bias design 
techniques to the specific nature of the recourse problem. Bailey (1995) demonstrates a 
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Central Composite Design (CCD) to highlight the effectiveness of LHS in reducing the 
variance of the estimator of E[h(x, co, T)]. The CCD has the advantage of simple 
implementation while still providing a large amount of information about the surface, but 
will not protect against model bias. 

For initial screening purposes, a small composite design is constructed using a 
resolution IV foldover of a Plackett-Burman augmented with 2 axial points for each 
parameter. The Plackett-Burman portion is used to estimate the largest possible number 
of terms in the fewest runs. This design screens only linear and pure quadratic effects, 
and reflects the assumption that a significant parameter will have either a significant 
linear or quadratic term. Second-order interactive terms are not considered in accordance 
with this assumption. Terms higher than second order are not addressed, an assumption 
we make under the sparsity of effects principle (Myers and Montgomery 1995). 
Furthermore, as the feasible space of the recourse problem is convex our use of second- 
order terms is appropriate. 

The choice of range for each decision variable is motivated by several factors. 
First, it must be large enough to provide the decision maker a practical and meaningful 
choice. Second, the range must be feasible, e.g. a nonnegative decision variable with a 
center value of five cannot have a range of six. Third, empirical studies have shown a 
relatively high variance in the region of optimality (see Bailey 1995); therefore the range 
must induce a sufficiently large change in the response to differentiate the effect of a 
variable from the effect of random variance (both system variance and estimator 
variance). Finally and conversely, the range must be sufficiently small to allow an 
accurate second-order polynomial approximation of the surface in the local region of 
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optimality, the region of interest. For each problem, the same factor range is used for 
each factor in every design to allow for unbiased comparisons, e.g if one factor has a 
range of ± 5 all factors have a range of ± 5. 

Several options exist for modeling screened factors in the reduced factor design. 
First, these variables can be held to a coded value of zero. Second, we can group these 
variables, either randomly, or by an external criteria, and vary them as a group across the 
design. Third, we can allow each screened variable to independently vary across the 
design using an assigned distribution. The first and third alternatives provide insight into 
the accuracy of the individual screening, while the second option provides knowledge on 
the group effects. 

After screening, we determine the center of the design space, construct the design 
for the first-stage variables, and fit the polynomial approximation of the problem. The 
resulting metamodel can then predict and provide sensitivity analysis within the design 
space (see Box and Draper 1987, Neter et al. 1996). 

We adopt Box and Draper’s (1987) minimum bias design, which is near-exact 
when used to minimize the bias of a second-order model of a cubic surface. Our design 
is adapted from a resolution V central composite design. We achieve minimum bias by 
scaling the design parameters such that the design moments, M y , equal the region 

moments, fi y , (a single subscripted moment denotes a pure moment, a double subscript is 

an interactive moment). For the second-order design, with the origin defined as the 
design center, all region moments through order 5 are zero, with the exception of the 
following: 
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Mi, = | w(x)xfdx 
o 

Ma, = j w(x)x*dx 
o 

Mu, =\w(x)x]x 2 j dx 
o 


where 


WW = 7^ 

P 

0 

(Box and Draper 1987, Myers and Montgomery 1995). 

Over a coded cuboidal design space the region of operability O is defined by the 
[-1,1] range of each coded variable. The weight function w(x) is 


w(*) = 



2 ' 


which is the inverse of the volume of the /-dimensional hypercube with side length 2. 

The limits of integration are constant and therefore interchangeable, resulting in the 
following values for the region moments (complete computations are shown in Appendix 
A) 



M4 = 


l 

5 



The relevant design moments are represented as 
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1 N 

m 2 =— f >, 2 
2> nU 


i n 

m '. = nZ x ? 

1 v n -1 


M. 


22 , 


1 N 

= -Yx 2 x 

N£i ' ■ 


where N = number of total design points. Equating the design moments and region 
moments results in: 

n f a 2 + 2a 2 a 2 = 7V73 
n.fct + 2a 4 a A = N/5 
nfa* = N 19 
n c + rif+ 2k — N 

where n/= the number of factorial design points, a = half length of one edge of the 
minimal bias hypercube, k = number of axial points, a = multiplier for the axial points, 
and n c = number of center points (Box and Draper 1987). The design parameters are not 
always integer, but the general form of the design is robust to inexactness in the 
parameters (Myers and Montgomery 1995). The use of any balanced design for the 
± sign matrix guarantees that the odd moments through order 5 are zero. Axial points 
contribute to the estimation of the pure quadratic terms, and multiple center points 
facilitate estimation of pure error and lack of model fit. 

The center of the design space must be chosen to provide the richest sensitivity 
analysis possible. Therefore, we center the design in such a way that the extremes of the 
cuboidal region have a roughly equal Z(x) value, exploiting this “flatness” for our 
sensitivity analysis. We do not move more than 0.6 coded units away from the optimal 
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for any parameter, and only move parameters with statistical significance in order to 
retain the validity of the screening design. While a rectangular region (not restricted to 
cuboidal) could encompass an even “flatter” region, the coded parameter estimates would 
be more difficult to interpret. In all cases only uncoded parameter estimates are provided. 

We demonstrate three methods of centering the design. Our first method uses the 
gradient obtained at the optimal solution by projecting from the optimal solution out 
The gradient is calculated using the method shown by Ermoliev (1983,1988) where for 
the i‘ h realization of 6>and T the negative gradient is 

-V* =c + n t T 

and for x k the unbiased estimate of the negative gradient for N samples is 

1 N 

-v, =--Yv, . 
z * n£( lk 

Within the region of interest the upper and lower limits of the design variables are based 
on the gradient. This method requires sampling at only one point. 

Our second method of locating the design center uses the linear and quadratic 
estimates from the screening design. The polynomial approximation from the axial 
screening design is 

/(■*) = +X, P iXi + X ^ 

where 


dx 


(/(*)) = o 


for 


xk — —■, k— \ ,2,... ,i 

r\ 7 7 * ' 

ICCk 
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thus yielding a center point for the design. 

The last centering methodology centers the design at the optimal solution. The 
first two strategies reflect an interest in a location least susceptible to variation in the 
parameters. The third strategy reflects an interest in characterizing the response to 
variation in the parameters in the immediate vicinity of the optimal solution. 

3.2: Modified Simplex-Lattice Koshal 

Two of the three problems we examine in this work contain equality constraints. The 
Simplex-Lattice (SL) design is applicable when equality constraints in the problem 
preclude a standard CCD (see Myers and Montgomery 1995). The CCD requires that 
each variable be capable of independent movement; however, an equality constraint 
requires any increase in one variable be appropriately offset by decreasing one or more 
associated variables. For the generalized equality constraint 

2>,=i 

i =1 

a {q, m} simplex-lattice, for modeling q parameters in a polynomial of degree m, is 
formed by equally spacing m+1 points on a simplex such that 

1 2 

x i - 0,—,—,...,1, i = 1,2 ,...,q 

m m 

and then forming all ordered sets (x 1 ,x 2 ,...,x q ) which satisfy the equality constraint 

(Myers and Montgomery, 1995). The simplex vertices 

(1,0,...,0), (0,1,...,0), ...,(0,0,...,l) 
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estimate pure linear effects, and the remaining points estimate interactions. In general, a 
set with k nonzero elements estimates a -order interaction. For m=2, the fitted 
polynomial is 

/(*)=£&*/ 

1=1 i< j 

Estimating a constant term in the SL requires the a priori removal of one term to allow 
estimation of the intercept. 

The simplex-lattice models the response for the entire range of the equality 
constraint, and we assume the response for the range of equality constraints in this class 
of problems to be too complex to model with acceptable accuracy using a single 
polynomial. As with the CCD and the minimal-bias model, we seek to model only the 
region about the optimal solution. We restrict the model to the region of interest using a 
set of deviation variables d t such that 

/= 1 

where dj denotes the deviation of x t from the optimal solution. In this manner the design 
is restricted to the local region. Since no set (c,0,...,0),c * 0 can satisfy this constraint, 
no direct estimation of linear terms can be made. Additionally, any deviation variable 
equals the negative of the sum of the remaining variables, forcing a singularity in the 
regression matrix. We therefore augment the design with axial points at the extreme 
feasible values for each variable. Since estimating pure quadratic terms, or an intercept 
term, results in a singular regression matrix, the general model follows the standard SL 
form. The Koshal form of estimating interaction terms is used to limit the number of 
samples that must be made (See Myers and Montgomery 1995). 
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3.3: Validation: 


We evaluate the validity of our polynomial approximation in the following 
manner. We first apply the standard measures of adequacy (residual analysis, lack of fit, 
etc.) using the Shapiro-Wilk test (Neter et al. 1996) for normality, and the Levene test 
(Neter et al. 1996) for nonconstant variance. We then sample additional points within the 
design space, and compare the predicted values of the polynomial metamodel to the 
values provided by the original model of (1). 
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4.0: Problem Descriptions and Computational Results 
4.1: 20-Term Problem Description and Computational Results 

The first problem we address is 20-Term, a vehicle allocation problem between a 
central depot and 20 outlying terminals (hence the name 20-Term). 20-Term is ideal for 
our study as it is a large problem (63 decision variables and 40 random variables) and the 
optimum is known (Bailey 1995). The decision variables x, through x 2 , are trailers (600 
available), while x 22 through x 42 are tractors (400 available); together, these comprise the 
existing fleet, at a modeled cost of zero. A complete “vehicle” requires one tractor and 
one or two trailers. Variables X 43 through x 63 are rental units (one combined tractor- 
trailer) with a daily cost of 100, and constrained to a maximum of 10,000 rental units. 
These constraints are represented mathematically as 

= 600 

/=1 

f>,=400 

i=22 

63 

< 10,000 

i=43 

and represent the first stage of the problem (corresponding to (1)). The T matrix 
allocates the decision variables among the central depot and outlying terminals without 
any stochastic representation, and is expressed mathematically as 

-Xj-x j+42 =0,j = l,2,...,2l 

- Xj - x J+2l =0,y' = 22,23,... ,42 
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The two equality constraints lack the necessary degrees of freedom to construct a 
Central Composite Design (CCD). Consequently, xj and x 2 2 are used as dependent 
variables such that 

Jt,=600-f>, 

i-2 

*22 =400 -Z X |- 

i—23 

Removing xj and x 22 from consideration projects the modeled surface onto the surface 
representing the outlying terminals. The forty independent random variates represent 
supply and demand at each of the outlying terminals; the lack of supply and demand 
variables for the central depot support the use of the central location as a staging point for 
the outlying terminals. 

At the optimal solution there are thirty-eight nonzero parameters (Bailey 1995), 
including Xj and x 22 . All parameters with a zero value are a priori removed to allow 
movement in both directions, and a parameter range of five is used with 200 replications 
at each point to estimate the value. The parameter screening reduces the number of 
variables under consideration to thirteen (screening design results are in Appendix D). 

The optimal value of x 25 is too small to allow a spherical region CCD, so the space 
is modeled as a cuboidal region (face centered CCD). A 2y~ 5 design base gives 256 

factorial runs, 26 axial points, and 42 center points for a total of AK324 runs. The 
relatively large number of center replications stabilizes estimator variance and allows 
good estimation of lack of fit. We demonstrate all three centering strategies with the 
CCD. Summary statistics for the models are shown in Table 4-1. 
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Table 4-1: 20-Term CCD Summary S' 

tatistics 

Centering Strategy 

Optimal 

Quadratic 

Gradient 

Rsquare 

0.997 

0.991 

0.997 

Rsquare Adj 

0.996 

0.990 

0.996 

RMSE 

399.147 

224.154 

363.333 

ANOVA Significance 

<.001 

<.001 

<.001 

Lack of Fit 

<.001 

0.601 

<.001 

Residual Normality 

0.770 

0.299 

0.893 

Residual Variance 

0.062 

0.422 

0.094 

Max Response 

281570 

262869 

280543 

Min Response 

254657 

254684 

254778 

Std Dev(Response) 

6230.107 

2220.225 

6028.360 


The CCD summary statistics suggest a good approximation, with normally 
distributed, constant variance residuals. The optimal centered model indicates no lack- 
of-fit, and model significance is strong. The quadratic-centered model has a high lack-of- 
fit; the extremes of this design are well outside the range of the initial screening design, 
and screened variables could possibly be significant in the new region. However, good 
ANOVA model significance coupled with good residual analysis allows the analysis to 
proceed. The RMSE and the standard deviation of the response indicate the quadratic 
centering does produce the desired effect - a near-optimal location least susceptible to 
change in the parameters. The gradient centering provides the best all-around model, 
with the highest R 2 and a 10% reduction in RMSE over the optimal centered model, as 
well as good residual analysis and no indication of lack of fit. 

A canonical analysis of the optimal-centered 13-variable CCD provides 
sensitivity analysis for the fitted model (see Myers and Montgomery 1995, Box and 
Draper 1987). The cuboidal design is not rotatable, therefore the standard errors for the 
eigenvalues of the second-order model cannot be extracted directly from the regression. 
Instead, we apply the Bisgaard and Ankenman (1998) approach of rotating the decision 
variable values using the matrix of eigenvectors associated with the original matrix of 
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second-order terms, and fitting the model using the rotated data. In this model, the 
eigenvalues are the coefficients of the pure quadratic terms, and the standard error of the 
coefficients are used as the standard error of the eigenvalues. Table 4-2 contains the 
results of the canonical analysis, and Table 4-3 contains the 95% confidence intervals for 
the eigenvalues. The eigenvector corresponding to the largest eigenvalue represents the 
maxima ridge for the system, while the eigenvector corresponding to the smallest 
eigenvalue represents the minima ridge. There are no zero eigenvalues, indicating that 
there is no pure ridge system. 

Table 4-2: 20-Term Maximal and Minimal Ridge Eigenvectors 


Eigenvalue 

62.449 

7.875 

X4 

0.514 

-0.003 

XII 

0.217 

0.042 

X23 

0.012 

0.000 

X24 

-0.059 

-0.002 

X25 

-0.597 

0.000 

X31 

-0.192 

-0.059 

X32 

-0.315 

0.722 

X33 

-0.317 

-0.686 

X34 

-0.225 

0.039 

X36 

-0.026 

-0.021 

X39 

0.185 

0.000 

X43 

-0.082 

0.012 

X62 

0.034 

0.005 


Table 4-3: Eigenvalues of 20-Term CCD 


Eigenvalue Std Error Lower 95% Upper 95% 


62.449 

1.054 

60.382 

64.515 

61.669 

1.046 

59.619 

63.718 

54.470 

1.910 

50.727 

58.213 

47.203 

0.975 

45.292 

49.113 

40.490 

1.597 

37.359 

43.621 

37.399 

1.011 

35.418 

39.380 

36.266 

1.050 

34.209 

38.323 

35.490 

3.995 

27.660 

43.319 

34.460 

1.078 

32.348 

36.572 

26.075 

2.854 

20.480 

31.670 

23.501 

1.322 

20.909 

26.092 

18.094 

2.944 

12.323 

23.865 

7.875 

1.282 

5.362 

10.388 
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Figure 4-1 is a graph of the observed response along the minimal and maximal 
eigenvectors. The region of interest is dominated by second-order terms, and Figure 4-1 
demonstrates the accuracy of the second-order ridge analysis, with the maximal ridge 


observations dominating the minimal ridge observations at each point. 
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Figure 4-1: Observations along Maxima and Minima Ridge 
Eigenvectors (20-Term) 


The minimal bias model is constructed in much the same manner as the CCD. 

For the 256 factorial point CCD base, the design moment equations are satisfied with 
a=0.6, a=3, and 20 center runs, centered at the optimal solution. Our range of interest is 
± 5 units for each variable; unfortunately, the optimal value of x 2 s = 6 is too small to 
accommodate the range of movement required for the axial points. Consequently, x 2 $ is 
dropped from the minimal bias portion of experimentation. Table 4-4 lists the summary 
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statistics for the regression, and Figure 4-2 is a graph of the predicted response and the 
observed response. 

Table 4-4: 20-Term Minimal Bias Summary Statistics 


Centering Strategy 

Optimal 

Rsquare 

0.946 

Rsquare Adj 

0.939 

RMSE 

503.167 

ANOVA Significance 

<.0001 

Lack of Fit 

<.0001 

Residual Normality 

0 

Residual Variance 

<.0001 

Max Response 

264468.8 

Min Response 

254762.1 

Std Dev(Response) 

2036.162 



Response 

Figure 4-2: Minimal Bias Response by Predicted Response 

The minimal bias model has a slightly worse R 2 than the CCD, and a root mean 
squared error (RMSE) 20% higher than the CCD. The deletion of x 25 appears to have had 
a noticeable effect on the regression. The residuals also violate the model requirement of 
normal distribution. This is seen in Figure 4-2, where the predicted values and the 
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observed values in the lower portion of the graph are not on the line, indicating a region 
of poor prediction and high residual values. If there exists a set of factor levels which 
produce poor estimation and a peak in residual values, this could possibly be remedied by 
regressing using weighted least-squares instead of the standard least-squares procedure 
used (Neter et al. 1996). We instead recommend the use of the CCD as it allows the use 
of all significant variables. 

20-Term has two equality constraints and one inequality constraint, each with 
mutually exclusive parameters. This allows blocking the problem into three parameter 
groupings. We apply the modified SL to the first constraint, holding x 22 through x 63 at 
their optimal solution values and modeling all interactions within the x : through x 2 i 
block. As with the CCD, a range of ± 5 for each nonzero variable is modeled, with all 
zero valued parameters excluded. The remaining sixteen parameters are modeled with 
240 two-way interactive samples (two for each interaction), 16 axial points, 32 6-way 
interactive points to allow for estimation of error, and 12 center points to allow an 
estimate of lack of fit. Summary statistics are in Table 4-5, and the deviation matrix is in 
appendix P. Figure 4-3 is error plotted against response for the non-axial portion of the 
design. 

Table 4-5: 20-Term MSLK Summary Statistics 


Centering Strategy 

Optimal 

Rsquare 

0.976 

RSquare Adj 

0.974 

RMSE 

17417.16 

ANOVA Significance 

<.0001 

Lack of Fit 

<.0001 

Residual Normality 

<.0001 
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Figure 4-3: 20-Term MSLK Error by Response 


Results for this design are not promising. RMSE is high, the residual analysis 
shows a lack of normality, and nonconstant variance is visually apparent in Figure 4-3. 
The axial points are not in the local region, and appear to reduce the effectiveness of the 
model in the region of optimality. A possible remedy would be to regress using weighted 
least-squares, minimizing the impact of the axial points. No further analysis of this 
design will be pursued. 

The CCD family of designs produces consistently good models. For the optimal- 
centered design, the CCD is the obvious choice, and to provide a location least sensitive 
to parameter changes, we recommend using either the gradient-centered method, or the 
quadratic-centering method with a screening design at the design center to choose 
variables which will improve lack of fit. 


4-8 




As a final validation step, we generate 5,000 random points within the design 
space, and allow factors with an optimal value of zero to vary uniformly from zero to 
two, while all non-zero variables range uniformly ±5 from their optimal value. 

Predicted values are computed and the percentage errors plotted against the distance from 
the design center in Figure 4-4. The worst error was less than 8%, with an average error 
of 2.5%. 

8.000% 

6.000% 

4 . 000 % 

2 . 000 % 

o 

£ 0 . 000 % 

3S 

- 2 . 000 % 

- 4 . 000 % 

- 6 . 000 % 

- 8 . 000 % 

15 20 25 30 35 40 45 50 55 60 

Distance from Design Center 

Figure 4-4: 20-Term Validation Error by Distance (Uncoded Space) 

4.2: SSN Problem Descriptions and Computational Results 

The next problem analyzed is SSN, a multi-commodity network flow model with 
31 nodes, 89 links (decision variables), 86 demand pairs (stochastic), and 706 routes. A 
route is comprised of no more than 3 links. Each link may be assigned a capacity, with 
no more than 1,008 total units of capacity. This constraint is represented mathematically 
as 
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89 

£*,£1,008 

i=1 

and is the only decision variable constraint. The literature provides neither a description 
of the acronym SSN nor units for the objective function; neither are required for the 
analysis. 

At the optimal solution the constraint is active, a situation referred to as a binding 
multi-component constraint (Myers and Montgomery 1995). One possible design choice 
for this type of region is a computer-generated design (Myers and Montgomery 1995). 
We instead relax the constraint and center a CCD at the optimal solution, and impose the 
constraint on the final model. 

Ten replications of 400 runs each at the optimal solution provides an estimate of 
variance (approximately 1.9), and an estimate of the optimal solution (approximately 
10.04). The high variance prompts the use of 800 replications at each point. An initial 
screening design using a cuboidal region with decision variable range size of five (all 
factors with a value less than five are a priori removed) provided poor results - an 
indefinite fit (indicated by the presence of both positive and negative eigenvalues) in a 
known convex space (Wets 1966). The summary statistics for this design are in table 4- 
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Table 4-6: Summary Statistics for SSN 

Centering Strategy 
RSquare 
RSquare Adj 
RMSE 

Max Eigenvalue 
Min Eigenvalue 


Cuboidal Screening Design 

Optimal 

0.872204 

0.760947 

0.942029 

1.194027 

-0.97663 
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The range is not large enough to differentiate between random variation and variation 
induced by decision variable changes, and indicates the need for a larger decision 
variable range. The use of a cuboidal range must be abandoned and each factor range is 
determined as 

r t = min(10, x iop ) 

where r, is the factor range for the ith variable and x ii0p is the optimal solution value for 
the ith variable. All variables with a value of zero are not considered. 

At the optimal solution there are fifty-five nonzero parameters (Morton 1999). 

The factor screening indicates that including thirteen variables results in an acceptably 
high R 2 of 0.87, while the use of all fifty-five results in only a small increase in R 2 to 
0.95. Forty-one variables are statistically significant in the region of interest, and a 
complete model of the region would require all forty-one; computational considerations 
for this work restrict the analysis to fewer factors. Consequently thirteen factors are 
chosen using a stepwise screening procedure (summary statistics and parameter estimates 
for the design are in Appendix J). A 2' V V5 design base is used, with 256 factorial runs, 26 

axial points, and 18 center points for a total of jV= 300 runs. As with 20-Term, the 
relatively large number of center replications stabilizes estimator variance and allows 
good estimation of lack of fit. The alternate centering strategies push the design center 
out of the feasible region and are not used in this problem. 

The CCD summary statistics in Table 4-7 suggest a good approximation, with 
normally distributed, constant variance residuals. There is evidence to suggest lack of fit 
due to the high variance of the system, but the R 2 of 0.896, good ANOVA model 
significance, and excellent residual analysis allows the model analysis to proceed. 
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Table 4-7: SSN CCD Summary Statistics 


Centering Strategy 

Optimal 

Rsquare 

0.896 

Rsquare Adj 

0.884 

RMSE 

0.886 

ANOVA Significance 

<.0001 

Lack of Fit 

0.389 

Residual Normality 

0.514 

Residual Variance 

0.560 


As with 20-Term, canonical analysis of the CCD provides sensitivity analysis for 
the fitted model. The design is not rotatable, and the Bisgaard and Ankenman (1998) 
method of estimating the standard errors of the eigenvalues is used here as well. Table 
4-8 contains the results of the canonical analysis, and Table 4-9 contains the 95% 
confidence intervals for the eigenvalues. For brevity only the maximal and minimal ridge 
eigenvectors are included. Eight of the thirteen eigenvalues cannot be distinguished from 
zero at the 95 % confidence level, indicating pure second-order ridge systems within the 
design space. The eigenvalues are all near zero, indicating very low curvature within the 
design region, and sampling along the minimal and maximal ridge eigenvectors confirms 
this. Figure 4-5 is a plot of the observations along the maximal and minimal 
eigenvectors; the observations along the distinct eigenvectors yield indistinct results, 
affirming that a first-order analysis is sufficient for the region. 
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Table 4-8: 


SSN Maxima] 


and Minimal Ridge Eigenvectors 


Eigenvalue 

0.011 

-0.002 

X12 

-0.477 

0.002" 

XI7 

0.058 

0.036 

X19 

0.000 

0.523 

X21 

0.047 

0.000 

X24 

-0.014 

0.006 

X30 

-0.002 

-0.702 

X33 

0.393 

-0.018 

X40 

0.326 

0.005 

X41 

-0.112 

-0.005 

X43 

-0.226 

0.008 

X53 

-0.012 

-0.482 

X60 

0.038 

-0.005 

X64 

-0.664 

0.003 


Table 4-9: Eigenvalues of SSN CCD 


Eigenvalue 

Std Error 

Lower 95% 

Upper 95% 

0.011 

0.002 

0.008 

0.014 

0.008 

0.002 

0.005 

0.011 

0.003 

0.001 

0.002 

0.005 

0.003 

0.001 

0.001 

0.005 

0.003 

0.001 

0.001 

0.004 

0.001 

0.002 

-0.002 

0.005 

0.001 

0.001 

-0.001 

0.002 

0.001 

0.001 

-0.001 

0.003 

0.001 

0.002 

-0.003 

0.004 

0.000 

0.001 

-0.001 

0.002 

0.000 

0.002 

-0.004 

0.003 

0.000 

0.001 

-0.002 

0.002 

-0.002 

0.001 

-0.003 

0.000 
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Distance from Design Center 


Figure 4-5: Observations along Maxima and Minima 
Ridge Eigenvectors (SSN) 

The first-order direction of greatest change, maxima, is represented by the vector 
of linear estimates (Myers and Montgomery 1995); any orthogonal vector is the direction 
of least change, minima, (for L the vector of linear estimates, P an alternative direction 
vector, LP=0 is the smallest magnitude of change possible; therefore P orthogonal to L is 
a direction of least change). Table 4-10 contains the maxima vector and a minima vector 
(other minima vectors exist; any vector orthogonal to the maxima is a minima vector), 
and Figure 4-6 is a plot of the observations along the maxima and minima vectors. 
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Table 4-10: Maxima and Minima First-Order Vectors for SSN CCD 


Variable 

Maxima 

Minima 

X12 

0.000 

0.000 

XI7 

-0.355 

0.500 

X19 

-0.106 

-0.500 

X21 

-0.539 

-0.500 

X24 

-0.078 

0.500 

X30 

-0.059 

0.500 

X33 

-0.196 

0.500 

X40 

-0.267 

-0.500 

X41 

-0.106 

0.500 

X43 

-0.131 

0.500 

X53 

-0.036 

-0.500 

X60 

-0.140 

0.500 

X64 

-0.114 

-0.500 



0 3 6 9 12 15 18 

Distance From Design Center 

Figure 4-6: Observations along First-Order Maxima and Minima (SSN) 

Figure 4-6 shows the accuracy of the direction vectors obtained from the CCD, 
and also demonstrates the highly variant nature of SSN. There is virtually no change in 
the minima direction, while the observations along the maxima show a clear rising 
tendency. 


4-15 





The use of only a small subset of the statistically significant parameters for the 
CCD preclude the type of whole-model random prediction validation used for 20-Term. 

4.3: Empirical Conclusions 

The empirical models of 20-Term and SSN provide an easily computable estimate 
for a decision maker. The methodology presented in Chapter 2 is validated by the 
accuracy of the models, in particular the sensitivity analysis. 
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5.0: Contributions and Future Research 

This research extends the methodology for characterizing local regions of the 
two-stage stochastic linear programming with recourse family of problems, using 
empirical modeling techniques. We provide a framework to accurately characterize the 
surface with a low-order polynomial and provide sensitivity analysis in the optimal 
region. We also demonstrate the utility of response surface methodology for this class of 
problems, and show the flexibility of RSM through the use of several different techniques 
to adapt to specific problems. 

While no alphabet-optimal designs were used in this work, the literature suggests 
the G and Q-optimal family of designs may prove useful in the characterization of this 
class of problems, since these designs specifically seek to stabilize prediction variance 
within the design region. The binding multi-component constraint in SSN could be 
modeled using a computer-generated design to produce a more accurate characterization 
of the optimal region than the CCD. Our approach of relaxing the binding constraint is 
not always a realistic possibility, and, in the case of a binding, non-releasable constraint, 
a computer-generated design could provide the necessary structure to characterize the 
surface. The nature of the specific problems studied led to the use of cuboidal design 
regions. A spherical design region is generally more desirable, and for problems where 
this is feasible, will most likely lead to more accurate prediction within the design region. 
In particular, a spherical design would allow the use of a minimum-bias design with all 
significant variables. An exploration of the minimum-bias design on a problem with the 
necessary freedom of range is valuable in minimizing error within the design region. 
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High variance hindered the analysis of SSN. Total variance is the sum of 
estimator variance and system variance, and although nothing can be done to minimize 
system variance, the total variance can be minimized through the use of estimator 
variance reduction techniques. While Latin hypercube sampling is used throughout this 
paper for this purpose, other possible techniques include antithetic variates and control 
variates. Furthermore, LHS requires independent variables, which are assumed to be 
present for each problem studied; however, for correlated variables LHS may not prove 
suitable. A comparative study of several different variance reduction techniques under 
different correlation assumptions should be undertaken on high-variance problems as 
SSN. 

Finally, statistical significance of variables was a key factor in screening factors 
within the design range. The relatively few number of significant variables in 20-Term 
led to the highly accurate characterization of the surface with a small polynomial, but 
with SSN the number of variables had to be restricted due to the large number of 
variables that were statistically significant. The use of a small-composite design could 
possibly result in a higher-accuracy design by allowing the efficient modeling of more 
factors with a manageable number of runs. 
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APPENDIX A: Terminology. 

We use the following concepts and terminology throughout this paper. 

Coded Variable: A variable transformed by scaling the minimum and maximum 
to -1 and 1, respectively (see Myers and Montgomery 1995, Box and Draper 
1987). See Uncoded Variable . 

Cuboidal region: A special case of the rectangular region where the range of 
each parameter is equal. See Rectangular Region . 

Deterministic Model: A model whose inputs are known with certainty (Banks et 
al. 1996). Contrast with Stochastic model. 

Extreme Point: Given a function /(jc) and a point x* in the domain of f{x), any 
s>0, and an s, if V s>0 and all s such that |v|j < e, either: 

1. f{x+s)<f{x) 

2. f(x* +s)>f(x*) 

Then x* is an extreme point of f. In case (1) above, x* is a local maximum, and 
in case (2) x* is a local minimum (Schmidt and Davis 1981). 

Global Maximum: Given a set of all local maximum points (xi, X2, X3,..., x;) for 
a function f, the global maximum Xh defined to be x h = max[xi, x 2 , x 3 ,..., x,] 
(Schmidt and Davis 1981). See Extreme Point . 

Global Minimum: Given a set of all local minimum points (xi, x 2 , x 3 ,..., Xi) for 
a function f, the global minimum x h defined to be x h = min[xi, x 2 , x 3 ,..., x,] 
(Schmidt and Davis 1981). See Extreme Point . 
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Karush-Kuhn-Tucker (KKT) conditions: A set of conditions necessary for a 
point to be an optimum (Bazaraa et al. 1990). KKT conditions are both necessary 
and sufficient for constrained optimization problems (Schmidt and Davis 1981). 
Random Number Generator: An algorithm for providing pseudo-random 
numbers. This paper uses a Linear Congruential Generator, where Z,^(aZ,. l + 
C)modulo m (Law and Kelton, 1991). 

Rectangular Region: A region where the range of each parameter is independent 
of all other parameters. See Cuboidd Region . 

Rotatable Design: A design in which the variance of the estimator is a function 
only of the distance from the design center (Box and Draper 1987). 

Stationary Point: See Extreme Point - An alternate definition is a point x* where 
the gradient of the function is zero (Winston 1994). 

Stochastic model: A model where at least one input is a random variable (Banks 
etal. 12). Contrast with Deterministic model. 

Uncoded Variable: A variable represented in original units. See Coded 
Variable. 
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APPENDIX B: Moment Computations. 
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APPENDIX C: Parabolic Design Centering Computations. 


f(x) = j3o + Y i ^i + y Z i aiXi 2 

SO 

■£-(/(*)) = P,+2a,x, 
ox i 

—(/( jc)) = /?. + 2a i x i = 0 (setting partial derivative with respect to Xi to zero) 

dx, 

/?, + 2 a i x j = 0 
2a i x j = -p t 
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APPENDIX D: 20-Term Screening Design Results. 


Response: Response 
Summary of Fit 


Source 

RSquare 

RSquare Adj 

Root Mean Square Error 

Mean of Response 

Observations (or Sum Wgts) 

Lack of Fit 

DF Sum of Squares 

0.912907 

0.900926 

2052.249 

275662.7 

216 

Mean Square 

F Ratio 

Lack of Fit 

98 

762895824 

7784651 

21.3888 

Pure Error 

91 

33120257 

363959 

Prob>F 

Total Error 

189 

796016081 


<.0001 



Parameter Estimates 



Term 

Estimate 

Std Error 

t Ratio 

Prob>|t| 

Intercept 

271170.32 

174.2928 

1555.8 

0.0000 

4 

-623.3938 

237.4169 

-2.63 

0.0094 

4*4 

1867.2617 

1312.017 

1.42 

0.1563 

11 

-423.4396 

237.4169 

-1.78 

0.0761 

11*11 

1592.502 

1312.017 

1.21 

0.2263 

23 

-1043.801 

237.4169 

-4.40 

<.0001 

23*23 

1562.801 

1312.017 

1.19 

0.2351 

24 

-547.6445 

237.4169 

-2.31 

0.0222 

24*24 

1056.07 

1312.017 

0.80 

0.4219 

25 

-1249.936 

237.4169 

-5.26 

<.0001 

25*25 

1962.9422 

1312.017 

1.50 

0.1363 

31 

-425.3867 

237.4169 

-1.79 

0.0748 

31*31 

1132.5585 

1312.017 

0.86 

0.3891 

32 

-660.2916 

237.4169 

-2.78 

0.0060 

32*32 

879.01807 

1312.017 

0.67 

0.5037 

33 

-764.1467 

237.4169 

-3.22 

0.0015 

33*33 

787.20664 

1312.017 

0.60 

0.5492 

34 

-611.2801 

237.4169 

-2.57 

0.0108 

34*34 

1423.7077 

1312.017 

1.09 

0.2792 

36 

-855.3613 

237.4169 

-3.60 

0.0004 

36*36 

1427.8315 

1312.017 

1.09 

0.2779 

39 

-509.739 

237.4169 

-2.15 

0.0331 

39*39 

1255.0798 

1312.017 

0.96 

0.3400 

43 

646.30283 

237.4169 

2.72 

0.0071 

43*43 

284.916 

1312.017 

0.22 

0.8283 

62 

425.7261 

237.4169 

1.79 

0.0745 

62*62 

-2245.31 

1312.017 

-1.71 

0.0887 
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APPENDIX E: 20-Term CCD Design Results. 


Response: Response 
Summary of Fit 



RSquare 


0.996531 



RSquare Adj 


0.995895 



Root Mean Square Error 

399.1465 



Mean of Response 


264883 



Observations (or Sum Wgts) 

324 




Lack of Fit 



Source 

DF 

Sum of Squares 

Mean Square 

F Ratio 

Lack of Fit 

232 

41139976 

177327 

3.0888 

Pure Error 

41 

2353825 

57410 

Prob>F 

Total Error 

273 

43493801 


<0001 


Source 

DF 

Analysis of Variance 

Sum of Squares 

Mean Square 

F Ratio 

Model 

50 

1.24935e10 

2.4987e8 

1568.374 

Error 

273 

43493801.2 

159318 

Prob>F 

C Total 

323 

1.2537e+10 


<.0001 



Parameter Estimates 



Term 

Estimate 

Std Error 

t Ratio 

Prob>|t| 

Intercept 

1503610.2 

1591824 

0.94 

0.3457 

X11*X23 

2.6107684 

0.997866 

2.62 

0.0094 

X11*X24 

3.37054 

0.997866 

3.38 

0.0008 

X23*X24 

25.832694 

0.997866 

25.89 

<.0001 

X24*X25 

7.0478868 

0.997866 

7.06 

<.0001 

X11*X31 

-2.779321 

0.997866 

-2.79 

0.0057 

X11*X32 

-9.124528 

0.997866 

-9.14 

<.0001 

X31*X32 

18.751966 

0.997866 

18.79 

<.0001 

X11*X33 

-6.008188 

0.997866 

-6.02 

<.0001 

X31*X33 

16.026274 

0.997866 

16.06 

<.0001 

X32*X33 

41.729022 

0.997866 

41.82 

<.0001 

X11*X34 

-1.975712 

0.997866 

-1.98 

0.0487 

X31*X34 

4.223032 

0.997866 

4.23 

<.0001 

X32*X34 

11.815232 

0.997866 

11.84 

<.0001 

X33*X34 

15.597613 

0.997866 

15.63 

<.0001 

X34*X36 

2.4440648 

0.997866 

2.45 

0.0149 

X25*X39 

-14.03803 

0.997866 

-14.07 

<.0001 

XI 1*X43 

-6.974411 

0.997866 

-6.99 

<.0001 

X4*X11 

8.8046915 

0.997866 

8.82 

<.0001 

X4*X23 

2.9062381 

0.997866 

2.91 

0.0039 

X4*X25 

-15.72449 

0.997866 

-15.76 

<.0001 

X4 

990.65336 

489.5964 

2.02 

0.0440 

X4*X4 

64.08403 

10.84804 

5.91 

<.0001 

XII 

144.56282 

832.3194 

0.17 

0.8622 

xirxn 

38.240406 

10.84804 

3.53 

0.0005 

X23 

-2377.273 

285.937 

-8.31 

<.0001 

X23*X23 

44.455534 

10.84804 

4.10 

<.0001 

X24 

-2160.368 

737.1017 

-2.93 

0.0037 

X24*X24 

36.71196 

10.84804 

3.38 

0.0008 

X25 

-888.8892 

135.2441 

-6.57 

<.0001 

X25*X25 

68.839778 

10.84804 

6.35 

<.0001 

X31 

-1482.89 

307.4997 

-4.82 

<.0001 

X31*X31 

22.03542 

10.84804 

2.03 

0.0432 

X32 

-2014.012 

393.7514 

-5.11 

<.0001 

X32*X32 

19.13702 

10.84804 

1.76 

0.0788 
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X33 

-2206.076 

X33*X33 

21.069174 

X34 

-2459.035 

X34*X34 

43.18785 

X36 

-2015.673 

X36*X36 

44.784666 

X39 

-1548.626 

X39*X39 

42.191168 

X62 

6933.9054 

X62*X62 

31.328842 

X4*X43 

-8.20122 

X43 

-6144.206 

X43*X43 

9.3969059 

X43*X62 

-22.96189 

X32*X36 

1.7881153 

X24*X43 

-1.879555 


436.3253 

-5.06 

<.0001 

10.84804 

1.94 

0.0531 

440.3417 

-5.58 

<.0001 

10.84804 

3.98 

<.0001 

413.1288 

-4.88 

<.0001 

10.84804 

4.13 

<.0001 

368.9154 

-4.20 

<.0001 

10.84804 

3.89 

0.0001 

4386.128 

1.58 

0.1151 

10.84804 

2.89 

0.0042 

0.997866 

-8.22 

<.0001 

8316.241 

-0.74 

0.4607 

10.84806 

0.87 

0.3871 

11.33358 

-2.03 

0.0437 

0.997866 

1.79 

0.0743 

0.997866 

-1.88 

0.0607 



APPENDIX F: 20-Term Minimal Bias Design Results. 


Response: Response 
Summary of Fit 



RSquare 



0.946082 



RSquare Adj 



0.938934 



Root Mean Square Error 


503.1673 



Mean of Response 


259413.9 



Observations (or Sum Wgts) 


300 




Lack of Fit 



Source 

DF 

Sum of Squares 

Mean Square 

F Ratio 

Lack of Fit 

245 


65842343 

268744 

5.1243 

Pure Error 

19 


996462 

52445 

Prob>F 

Total Error 

264 


66838806 


<.0001 

Max RSq 






0.9992 








Parameter Estimates 



Term 


Estimate 

Std Error 

t Ratio 

Prob>|t| 

intercept 


3028128.5 

638197.3 

4.74 

<.0001 

X4*X11 


10.782175 

3.494217 

3.09 

0.0022 

X23*X24 


20.892113 

3.494217 

5.98 

<.0001 

X11*X32 


-12.25127 

3.494217 

-3.51 

0.0005 

X31*X32 


13.247014 

3.494217 

3.79 

0.0002 

X31*X33 


9.4100612 

3.494217 

2.69 

0.0075 

X32*X33 


31.499883 

3.494217 

9.01 

<.0001 

X32*X34 


12.065418 

3.494217 

3.45 

0.0006 

X33*X34 


10.777376 

3.494217 

3.08 

0.0023 

X4*X43 


-7.401196 

3.494217 

-2.12 

0.0351 

X11*X43 


-8.71441 

3.494217 

-2.49 

0.0132 

X34*X43 


-10.41168 

3.494217 

-2.98 

0.0032 

X4 


754.70084 

1349.064 

0.56 

0.5763 

X4*X4 


58.513029 

4.337515 

13.49 

<.0001 

XII 


224.19558 

1372.754 

0.16 

0.8704 

X11*X11 


46.270288 

4.337515 

10.67 

<.0001 

X23 


-2397.806 

151.9513 

-15.78 

<.0001 

X23*X23 


59.700475 

4.337515 

13.76 

<.0001 

X24 


-2797.826 

255.8447 

-10.94 

<.0001 

X24*X24 


41.13873 

4.337515 

9.48 

<.0001 

X31 


-1760.195 

153.924 

-11.44 

<.0001 

X31*X31 


42.366866 

4.337515 

9.77 

<.0001 

X32 


-2141.127 

225.3012 

-9.50 

<.0001 

X32*X32 


37.587771 

4.337515 

8.67 

<.0001 

X33 


-2691.638 

203.5631 

-13.22 

<.0001 

X33*X33 


40.46728 

4.337515 

9.33 

<.0001 

X34 


1087.8992 

1352.794 

0.80 

0.4220 

X34*X34 


56.07097 

4.337515 

12.93 

<.0001 

X36 


-2218.545 

165.1367 

-13.43 

<.0001 

X36*X36 


53.186959 

4.337515 

12.26 

<.0001 

X39 


-2089.885 

147.8232 

-14.14 

<.0001 

X39*X39 


57.564489 

4.337515 

13.27 

<.0001 

X43 


-13557.83 

3325.77 

-4.08 

<.0001 

X43*X43 


18.458021 

4.337515 

4.26 

<.0001 

X62 


-1634.208 

251.7799 

-6.49 

<.0001 

X62*X62 


27.616426 

4.337515 

6.37 

<.0001 
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APPENDIX G: 20-Term Gradient Centered CCD Results. 


Response: Response 
Summary of Fit 


RSquare 0.99693 

RSquare Adj 0.996367 

Root Mean Square Error 363.3332 

Mean of Response 264537.2 

Observations (or Sum Wgts) 324 


Analysis of Variance 


Source 

DF 

Sum of Squares 

Mean Square 

F Ratio 

Model 

50 

1.17021e10 

2.3404e8 

1772.905 

Error 

273 

36038998.5 

132011 

Prob>F 

C Total 

323 

1.17382e10 


0.0000 




Lack of Fit 



Source 

DF 

Sum of Squares 

Mean Square 

F Ratio 

Lack of Fit 

232 


34194797 

147391 

3.2768 

Pure Error 

41 


1844201 

44981 

Prob>F 

Total Error 

273 


36038998 


<.0001 

Max RSq 






0.9998 








Parameter Estimates 



Term 


Estimate 

Std Error 

t Ratio 

Prob>|t| 

Intercept 


1336418.5 

1471973 

0.91 

0.3647 

X11*X23 


1.7409361 

0.908333 

1.92 

0.0563 

X11*X24 


2.9920102 

0.908333 

3.29 

0.0011 

X23*X24 


25.840941 

0.908333 

28.45 

<.0001 

X24*X25 


4.5274713 

0.908333 

4.98 

<.0001 

X11*X31 


-3.444468 

0.908333 

-3.79 

0.0002 

X11*X32 


-8.610768 

0.908333 

-9.48 

<.0001 

X31*X32 


18.925932 

0.908333 

20.84 

<.0001 

X11*X33 


-4.835923 

0.908333 

-5.32 

<.0001 

X31*X33 


15.411596 

0.908333 

16.97 

<.0001 

X32*X33 


41.627596 

0.908333 

45.83 

<.0001 

X11*X34 


-2.273658 

0.908333 

-2.50 

0.0129 

X31*X34 


3.2788931 

0.908333 

3.61 

0.0004 

X32*X34 


10.595318 

0.908333 

11.66 

<.0001 

X33*X34 


12.513428 

0.908333 

13.78 

<.0001 

X34*X36 


2.4008147 

0.908333 

2.64 

0.0087 

X25*X39 


-14.62376 

0.908333 

-16.10 

<.0001 

X11*X43 


-7.266431 

0.908333 

-8.00 

<.0001 

X4*X11 


8.9549337 

0.908333 

9.86 

<.0001 

X4*X23 


1.4941426 

0.908333 

1.64 

0.1011 

X4*X25 


-14.68016 

0.908333 

-16.16 

<.0001 

X4 


808.76588 

447.8614 

1.81 

0.0720 

X4*X4 


65.130452 

9.874699 

6.60 

<.0001 

XII 


-206.4181 

776.4456 

-0.27 

0.7906 

X11*X11 


44.323444 

9.874699 

4.49 

<.0001 

X23 


-2551.273 

260.3907 

-9.80 

<.0001 

X23*X23 


52.752642 

9.874699 

5.34 

<.0001 

X24 


-1374.363 

672.4246 

-2.04 

0.0419 

X24*X24 


26.036126 

9.874699 

2.64 

0.0089 

X25 


-770.189 

123.1093 

-6.26 

<.0001 

X25*X25 


64.319268 

9.874699 

6.51 

<.0001 

X31 


-1656.424 

280.0714 

-5.91 

<.0001 

X31*X31 


29.98537 

9.874699 

3.04 

0.0026 

X32 


-2021.564 

358.5487 

-5.64 

<.0001 
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X32*X32 

19.56145 

X33 

-2269.759 

X33*X33 

23.393128 

X34 

-2094.686 

X34*X34 

33.193372 

X36 

-1657.843 

X36*X36 

35.352226 

X39 

-1624.574 

X39*X39 

44.522032 

X62 

-3317.159 

X62*X62 

20.937604 

X4*X43 

-7.79008 

X43 

-4501.752 

X43*X43 

6.1805064 

X43*X62 

5.2905849 

X32*X36 

1.7477877 

X24*X43 

-2.238108 


9.874699 

1.98 

0.0486 

397.2903 

-5.71 

<.0001 

9.874699 

2.37 

0.0185 

420.3747 

-4.98 

<.0001 

9.874699 

3.36 

0.0009 

376.1059 

-4.41 

<.0001 

9.874699 

3.58 

0.0004 

335.8145 

-4.84 

<.0001 

9.874699 

4.51 

<.0001 

4026.074 

-0.82 

0.4107 

9.874699 

2.12 

0.0349 

0.908333 

-8.58 

<.0001 

7629.684 

-0.59 

0.5557 

9.874715 

0.63 

0.5319 

10.31668 

0.51 

0.6085 

0.908333 

1.92 

0.0554 

0.908333 

-2.46 

0.0144 


APPENDIX H: 20-Term Quadratic Centered CCD Results. 


Response: Response 
Summary of Fit 



RSquare 


0.991196 



RSquare Adj 


0.989807 



Root Mean Square Error 

224.1543 



Mean of Response 


259082.7 



Observations (or Sum Wgts) 

324 




Analysis of Variance 



Source 

DF 

Sum of Squares 

Mean Square 

F Ratio 

Model 

44 

1578177439 

35867669 

713.8532 

Error 

279 

14018400.4 

50245.16 

Prob>F 

C Total 

323 

1592195839 


<.0001 




Lack of Fit 



Source 

DF 

Sum of Squares 

Mean Square 

F Ratio 

Lack of Fit 

238 


11873393 

49888.2 

0.9536 

Pure Error 

41 


2145008 

52317.3 

Prob>F 

Total Error 

279 


14018400 


0.6009 

Max RSq 






0.9987 








Parameter Estimates 



Term 


Estimate 

Std Error 

t Ratio 

Prob>|t| 

Intercept 


2321042.2 

904587.7 

2.57 

0.0108 

X23*X24 


3.3108589 

0.560386 

5.91 

<.0001 

X24*X25 


1.1674145 

0.560386 

2.08 

0.0381 

X11*X32 


-4.537755 

0.560386 

-8.10 

<.0001 

X31*X32 


1.8085418 

0.560386 

3.23 

0.0014 

XI 1*X33 


-3.024751 

0.560386 

-5.40 

<.0001 

X31*X33 


1.0378902 

0.560386 

1.85 

0.0651 

X32*X33 


5.4214124 

0.560386 

9.67 

<.0001 

X31*X34 


-1.473253 

0.560386 

-2.63 

0.0090 

X32*X34 


1.7675876 

0.560386 

3.15 

0.0018 

X33*X34 


2.8927503 

0.560386 

5.16 

<.0001 

X23*X36 


-2.010192 

0.560386 

-3.59 

0.0004 

X23*X39 


1.4024652 

0.560386 

2.50 

0.0129 

X11*X43 


-7.640161 

0.560386 

-13.63 

<.0001 

X36*X62 


1.0538908 

0.560386 

1.88 

0.0611 

X4*X11 


9.4314731 

0.560386 

16.83 

<.0001 

X4*X25 


-8.148226 

0.560386 

-14.54 

<.0001 

X4*X43 


-8.056264 

0.560386 

-14.38 

<.0001 

X4*X62 


1.0557426 

0.560386 

1.88 

0.0606 

X4 


1463.2532 

284.3831 

5.15 

<.0001 

X4*X4 


43.642317 

6.092085 

7.16 

<.0001 

XII 


708.48996 

478.5695 

1.48 

0.1399 

X11*X11 


31.963091 

6.092085 

5.25 

<.0001 

X23 


-569.0956 

196.4973 

-2.90 

0.0041 

X23*X23 


12.657967 

6.092085 

2.08 

0.0386 

X24 


-443.1499 

390.0391 

-1.14 

0.2569 

X24*X24 


5.3961512 

6.092085 

0.89 

0.3765 

X25 


-474.3244 

111.4665 

-4.26 

<.0001 

X25*X25 


26.269091 

6.092085 

4.31 

<.0001 

X31 


-308.4809 

208.2489 

-1,48 

0.1397 

X31*X31 


6.9440391 

6.092085 

1.14 

0.2553 

X32 


-233.067 

257.4288 

-0.91 

0.3661 

X32*X32 


3.8464511 

6.092085 

0.63 

0.5283 

X33 


202.88064 

281.613 

0.72 

0.4719 


H-l 



X33*X33 

-7.326207 

6.092085 

-1.20 

X34 

-1124.084 

268.8027 

-4.18 

X34*X34 

21.133287 

6.092085 

3.47 

X36 

-872.4175 

268.7081 

-3.25 

X36*X36 

18.322487 

6.092085 

3.01 

X39 

-227.1465 

243.8643 

-0.93 

X39*X39 

4.6256471 

6.092085 

0.76 

X43 

-10480.69 

4703.147 

-2.23 

X43*X43 

14.047911 

6.092095 

2.31 

X62 

-1018.463 

353.6669 

-2.88 

X62*X62 

16.274521 

6.092085 

2.67 


H-2 


0.2302 

<.0001 

0.0006 

0.0013 

0.0029 

0.3524 

0.4483 

0.0266 

0.0218 

0.0043 

0.0080 


APPENDIX I: 20-Term Modified Simplex-Lattice Koshal Design Results. 


Response: Response 
Summary of Fit 



RSquare 


0.975679 



RSquare Adj 


0.974213 



Root Mean Square Error 

17417.16 



Mean of Response 


282484.3 



Observations (or Sum Wgts) 

300 




Analysis of Variance 



Source 

DF 

Sum of Squares 

Mean Square 

F Ratio 

Model 

17 

3.43185e12 

2.019e11 

665.4640 

Error 

282 

8.55469e10 

3.0336e8 

Prob>F 

C Total 

299 

3.5174e+12 


<.0001 


Source 

DF 

Lack of Fit 

Sum of Squares 

Mean Square 

F Ratio 

Lack of Fit 

271 

8.55462e10 


3.1567e8 

5300.927 

Pure Error 

11 

655046.753 


59549.7 

Prob>F 

Total Error 

Max RSq 

1.0000 

Term 

282 

8.55469e10 

Parameter Estimates 
Estimate 

Std Error 

t Ratio 

<.0001 

Prob>|t| 

X1*X19 


-109.5599 

2.186617 

-50.10 

<.0001 

X2*X4 


-154.1413 

20.36459 

-7.57 

<.0001 

XI 


1254.7085 

28.95107 

43.34 

<.0001 

X2 


1229.013 

28.99592 

42.39 

<.0001 

X3 


1168.1931 

28.94112 

40.36 

<.0001 

X4 


1259.9121 

29.02028 

43.41 

<.0001 

X7 


1236.53 

28.98149 

42.67 

<.0001 

XI0 


1214.3189 

28.98044 

41.90 

<.0001 

XII 


1193.3451 

28.97309 

41.19 

<.0001 

XI2 


1186.906 

28.96554 

40.98 

<.0001 

XI3 


1205.3034 

28.9655 

41.61 

<.0001 

X14 


1207.0091 

28.97622 

41.66 

<.0001 

XI5 


1192.8421 

28.96919 

41.18 

<.0001 

XI6 


1216.4698 

28.97722 

41.98 

<.0001 

XI7 


1200.8016 

28.95631 

41.47 

<.0001 

X18 


1221.3663 

28.9741 

42.15 

<.0001 

XI9 


1219.8492 

29.01718 

42.04 

<.0001 

X21 


1191.1923 

28.97175 

41.12 

<.0001 
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APPENDIX J: SSN Screening Design Results. 


Response: Response 
Summary of Fit 

RSquare 
RSquare Adj 
Root Mean Square Error 
Mean of Response 
Observations (or Sum Wgts) 


0.866135 

0.856882 

1.598977 

14.25722 

233 


Analysis of Variance 


Source 

DF 

Sum of Squares 

Mean Square 

F Ratio 

Model 

15 

3589.7399 

239.316 

93.6024 

Error 

217 

554.8101 

2.557 

Prob>F 

C Total 

232 

4144.5500 


<.0001 




Lack of Fit 



Source 

DF 

Sum of Squares 

Mean Square 

F Ratio 

Lack of Fit 

129 


490.56735 

3.80285 

5.2092 

Pure Error 

88 


64.24276 

0.73003 

Prob>F 

Total Error 

217 


554.81011 


<.0001 

Max RSq 






0.9845 








Parameter Estimates 



Term 


Estimate 

Std Error 

t Ratio 

Prob>|t| 

Intercept 


42.271817 

3.342492 

12.65 

<.0001 

XI2 


-0.046289 

0.014626 

-3.16 

0.0018 

X17 


-0.045825 

0.014626 

-3.13 

0.0020 

X19 


-0.042661 

0.014626 

-2.92 

0.0039 

X21 


-0.062233 

0.014626 

-4.25 

<.0001 

X24 


-0.074361 

0.014626 

-5.08 

<.0001 

X30 


-0.043002 

0.014626 

-2.94 

0.0036 

X33 


-1.36179 

0.355138 

-3.83 

0.0002 

X33*X33 


0.0294913 

0.008064 

3.66 

0.0003 

X40 


-0.071224 

0.018283 

-3.90 

0.0001 

X41 


-0.053037 

0.014626 

-3.63 

0.0004 

X43 


-0.052874 

0.014626 

-3.61 

0.0004 

X53 


-1.035472 

0.179946 

-5.75 

<.0001 

X53*X53 


0.0562222 

0.009956 

5.65 

<.0001 

X60 


-0.05739 

0.020895 

-2.75 

0.0065 

X64 


-0.072194 

0.014626 

-4.94 

<.0001 
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APPENDIX K: SSN CCD Results. 


Response: Response 
Summary of Fit 


RSquare 0.896G45 

RSquare Adj 0.883586 

Root Mean Square Error 0.885606 

Mean of Response 12.93567 

Observations (or Sum Wgts) 300 


Analysis of Variance 


Source 

DF 

Sum of Squares 

Mean Square 

F Ratio 

Model 

32 

1805.0057 

56.4064 

71.9197 

Error 

267 

209.4074 

0.7843 

Prob>F 

C Total 

299 

2014.4130 


<.0001 


Source 

Lack of Fit 

DF Sum of Squares 

Mean Square 

F Ratio 

Lack of Fit 

250 

197.71777 

0.790871 

1.1501 

Pure Error 

17 

11.68963 

0.687625 

Prob>F 

Total Error 

267 

209.40739 


0.3891 

Max RSq 

0.9942 

Term 

Parameter Estimates 

Estimate Std Error 

t Ratio 

Prob>|t| 

Intercept 

34.404515 

0.91822 

37.47 

<0001 

17 

-0.354809 

0.091125 

-3.89 

0.0001 

19 

-0.106144 

0.020675 

-5.13 

<.0001 

21 

-0.539014 

0.094082 

-5.73 

<.0001 

24 

-0.078173 

0.007813 

-10.01 

<.0001 

30 

-0.059246 

0.012661 

-4.68 

<.0001 

33 

-0.195804 

0.027095 

-7.23 

<.0001 

40 

-0.266675 

0.041196 

-6.47 

<.0001 

41 

-0.10639 

0.009575 

-11.11 

<.0001 

43 

-0.130725 

0.01549 

-8.44 

<.0001 

53 

-0.036229 

0.023781 

-1.52 

0.1288 

60 

-0.139839 

0.02028 

-6.90 

<.0001 

17*17 

0.0125461 

0.004491 

2.79 

0.0056 

21*21 

0.0158001 

0.004491 

3.52 

0.0005 

12*40 

0.0011789 

0.000692 

1.70 

0.0896 

12*64 

-0.003125 

0.000326 

-9.60 

<.0001 

19*30 

0.001031 

0.000554 

1.86 

0.0636 

21*33 

-0.001002 

0.000554 

-1.81 

0.0715 

33*40 

0,0015349 

0.000692 

2,22 

0.0274 

21*41 

0.0025412 

0.000554 

4.59 

<.0001 

40*41 

0,0014065 

0.000692 

2.03 

0.0431 

21*43 

0.0021281 

0.000554 

3.84 

0.0002 

33*43 

0.0011151 

0.000554 

2.01 

0.0449 

40*43 

0.001925 

0.000692 

2.78 

0.0058 

17*53 

0.0010512 

0.000615 

1.71 

0.0886 

30*53 

-0.000895 

0.000615 

-1.45 

0.1469 

21*60 

0.0048547 

0.000917 

5.29 

<.0001 

24*60 

0.0019155 

0.000791 

2.42 

0.0161 

40*60 

-0.001205 

0.000988 

-1.22 

0.2240 

17*64 

0.0015133 

0.000554 

2.73 

0.0067 

33*64 

0.0033522 

0.000554 

6.06 

<.0001 

40*64 

0.0016323 

0.000692 

2.36 

0.0190 

64 

-0.114424 

0.016932 

-6.76 

<.0001 
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APPENDIX L: Random Number Generator Code and Random Array Arranging Code (C 

Programming Language). 

Random number generator (See Banks et al. 1996) 

{ 

randl = (seed 1*40014)%2147483563; 

if(randl < 0) randl = 2147483563 + randl; 
rand2 = (seed2*40692)%2147483399; 

if(rand2 < 0) rand2 = 2147483399 + rand2; 
rand3 = (randl - rand2)%2147483562; 

if(rand3 < 0) rand3 = 2147483562 + rand3; 
seedl = randl; 
seed2 = rand2; 
return rand3; 

} 


Random array ordering procedure for 400 sample points 

{ 

num = 400; 

/♦Fill the arrays*/ 
for(il = 0; il <= 399; il++) 

{ 

ordervec[il] = il; /*Produces array of integers from 0 to 399*/ 
randomvec[il] = 0; /*Produces array of zeros*/ 

} 

for(temp = 0; temp <= 399; temp++) 

{ 

/*Generates random integer between 0 & num-1*/ 
i2 = genl(i2); 
i4 = i2%num; 

if(i4 < 0) i4 = num + i4; 

/♦Places the number from ordervec[i4] into randomvecftemp]*/ 
randomvecftemp] = ordervec[i4]; 

/♦Moves every element in ordervec (below the number drawn) up 1 */ 
for(i3 = i4; i3 <= num-1; i3++) 

{ 

if(i3 !=399) ordervec[i3] = ordervec[i3+l]; 

} 

/♦Decrements num, but never to zero*/ 
if(num >1) num~; 

} 

return; 

} 
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APPENDIX M: Sampling Pseudocode. 


Initialize: 

For each random variate, n samples at each point, need three arrays of size w, one with the 
stochastic values and two of zeros. The number of each stochastic value of probability p is found by 
multiplying p*n. Non-integer values are dealt with using the McKay 1988 procedure (also shown in Bailey 
I 995 ). 

A file of decision variable values (decval.prn) is produced, with each row containing a complete 
set of values to sample space. A separate file (mum.prn) contains random number seeds and number of 
samples (rows) in decision variable file. 

Sampling: 

Read MPS problem input file into solver 

Read random number seeds and number of samples from mum.prn. 

Assign i ~ number of samples 

For loop counter = 0 to loop counter = i-1 

Read next row of decision variable values from decval.prn 
Change decision variable values in solver 

For loop counter 2 = 0 to loop counter 2 = n-l 

Arrange random vector (see Appendix L) 

Arrange stochastic vectors in random order of random vector 

Change stochastic values in solver 

Optimize problem with current values 

Store solution value 

End 

Compute average of values 
Print average to file 

End 

End 

End 

End 
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APPENDIX N: 20-Term CCD Design Matrix. 



N-l 


44 1 

45 1 

46 1 

47 1 

48 1 

49 1 

50 1 

51 1 

52 1 

53 1 

54 1 

55 1 

56 1 

57 1 

58 1 

59 1 

60 1 

61 1 

62 1 

63 1 

64 1 

65 1 

66 1 

67 1 

68 1 

69 1 

70 1 

71 1 

72 1 

73 1 

74 1 

75 1 

76 1 

77 1 

78 1 

79 1 

80 1 

81 1 

82 1 

83 1 

84 1 

85 1 

86 1 

87 1 

88 1 

89 1 

90 1 

91 1 

92 1 

93 1 


1 -1 

1 -1 

1 -1 

1 -1 

1 -1 

1 -1 

1 -1 

1 -1 

1 -1 

1 -1 

1 -1 

1 -1 

1 -1 

1 -1 

1 -1 

1 -1 

1 -1 

1 -1 

1 -1 

1 -1 

1 -1 

-1 1 

-1 1 

-1 1 

-1 1 

-1 1 

-1 1 

-1 1 

-1 1 

-1 1 

-1 1 

-1 1 

-1 1 

-1 1 

-1 1 

-1 1 

-1 1 

-1 1 

-1 1 

-1 1 

-1 1 

-1 1 

-1 1 

-1 1 

-1 1 

-1 1 

-1 1 

-1 1 

-1 1 

-1 1 


1-1 1-1 -1 

1 - 1-1 1 1 

1 - 1-1 1 -1 

1 -1 -1 -1 1 

1 -1 -1 -1 -1 

-11111 
- 1111-1 
- 111-11 
- 111-1 -1 
-11-111 
-11-11 -1 
-1 1 - 1-1 1 

-1 1 -1 -1 -1 

-1-1111 
- 1-111 -1 
-1 - 11-11 
-1 -1 1 -1 -1 

-1 - 1-111 
-1 -1 -1 1 -1 

-1 -1 -1 -1 1 

-1 -1 -1 -1 -1 

11111 
1111-1 
111-11 
1 1 1 - 1-1 

11-111 
1 1-1 1-1 

1 1 - 1-1 1 

1 1 - 1-1 -1 

1-1 1 1 1 

1-1 1 1-1 

1-1 1-1 1 

1-1 1-1 -1 

1 - 1-1 1 1 

1 -1 -1 1 -1 

1 -1 -1 -1 1 

1 -1 -1 -1 -1 

-11111 
- 1111-1 
- 111-11 
- 111-1 -1 
- 11-111 
- 11-11 -1 
-1 1 - 1-1 1 

-1 1 -1 -1 -1 

- 1-1111 
- 1-111 -1 
-1 - 11-11 
-1 -1 1 -1 -1 

-1 - 1-111 


-1 1 - 1-1 1 

-1 -1 1 -1 1 

-1 -1 -1 1 -1 

-1 -1 -1 -1 1 

- 1-111 -1 
1 1 1 - 1-1 
11-111 
1 1 - 1-1 -1 

11111 
1 - 1-1 1 1 

1-1 1-1 -1 

1-1111 
1 -1 -1 -1 -1 

1 -1 -1 -1 -1 

1-1111 
1-1 1-1 -1 

1 - 1-1 1 1 

11111 
1 1 - 1-1 -1 

11-111 
1 1 1 - 1-1 

- 111-11 
-1 1-1 1-1 

-1 1 - 1-1 1 

-1111-1 
-1 -1 -1 1 -1 

-1 - 11-11 
- 1-111 -1 
-1 -1 -1 -1 1 

-1 -1 -1 -1 1 

- 1-111 -1 
-1 - 11-11 
-1 -1 -1 1 -1 

- 1111-1 
-1 1 - 1-1 1 

- 11-11 -1 
- 111-11 
1-1111 
1 -1 -1 -1 -1 

1 - 1-1 1 1 

1-1 1-1 -1 

1 1 - 1-1 -1 

11111 
1 1 1 - 1-1 

11-111 
11-111 
1 1 1 - 1-1 

11111 
1 1 - 1-1 -1 

1-1 1-1 -1 
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94 

1 

-1 

1 -1 -1 

-1 

1 

-1 

1 

-1 

-1 

1 

1 

95 

1 

-1 

1 -1 -1 

-1 

-1 

1 

1 

-1 

-1 

-1 

-1 

96 

1 

-1 

1 -1 -1 

-1 

-1 

-1 

1 

-1 

1 

1 

1 

97 

1 

-1 

-1 

1 1 

1 

1 

1 

1 

-1 

1 

-1 

-1 

98 

1 

-1 

-1 

1 1 

1 

1 

-1 

1 

-1 

-1 

1 

1 

99 

1 

-1 

-1 

1 1 

1 

-1 

1 

1 

-1 

-1 

-1 

-1 

100 

1 

-1 

-1 

1 1 

1 

-1 

-1 

1 

-1 

1 

1 

1 

101 

1 

-1 

-1 

1 1 

-1 

1 

1 

1 

1 

-1 

1 

1 

102 

1 

-1 

-1 

1 1 

-1 

1 

-1 

1 

1 

1 

-1 

-1 

103 

1 

-1 

-1 

1 1 

-1 

-1 

1 

1 

1 

1 

1 

1 

104 

1 

-1 

-1 

1 1 

-1 

-1 

-1 

1 

1 

-1 

-1 

-1 

105 

1 

-1 

-1 

1 -1 

1 

1 

1 

1 

1 

-1 

-1 

-1 

106 

1 

-1 

-1 

1 -1 

1 

1 

-1 

1 

1 

1 

1 

1 

107 

1 

-1 

-1 

1 -1 

1 

-1 

1 

1 

1 

1 

-1 

-1 

108 

1 

-1 

-1 

1 -1 

1 

-1 

-1 

1 

1 

-1 

1 

1 

109 

1 

-1 

-1 

1 -1 

-1 

1 

1 

1 

-1 

1 

1 

1 

110 

1 

-1 

-1 
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-1 

1 

-1 
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-1 

-1 

-1 

-1 
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-1 

-1 
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-1 
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1 

1 
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-1 
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-1 
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1 

1 

1 

-1 
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-1 

-1 
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1 

1 

-1 

-1 

1 

-1 

-1 

1 

115 
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-1 

-1 
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1 

-1 

1 

-1 

1 

-1 

1 

-1 

116 

1 

-1 

-1 
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-1 

-1 

-1 
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-1 

1 
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-1 
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-1 
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-1 
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1 
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-1 

-1 

-1 
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-1 
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1 

-1 

-1 
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1 

-1 

-1 

-1 

1 

-1 

1 
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1 

-1 

-1 

1 -1 

1 

-1 

1 

-1 

-1 

1 

1 

-1 

124 

1 

-1 

-1 
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1 

-1 

-1 

-1 

-1 

-1 

-1 

1 
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1 

-1 

-1 

1 -1 

-1 

1 

1 

-1 

1 

1 

-1 

1 

126 

1 

-1 

-1 

1 -1 

-1 

1 

-1 

-1 

1 

-1 

1 

-1 

127 

1 

-1 

-1 

1 -1 

-1 

-1 

1 

-1 

1 

-1 

-1 

1 

128 

1 

-1 

-1 

1 -1 

-1 

-1 

-1 

-1 

1 

1 

1 

-1 

129 

-1 

1 

1 

1 1 

1 

1 

1 

-1 

1 

1 

1 

-1 

130 

-1 

1 

1 

1 1 

1 

1 

-1 

-1 

1 

-1 

-1 

1 

131 

-1 

1 

1 

1 1 

1 

-1 

1 

-1 

1 

-1 

1 

-1 

132 

-1 

1 

1 

1 1 

1 

-1 

-1 

-1 

1 

1 

-1 

1 

133 

-1 

1 

1 

1 1 

-1 

1 

1 

-1 

-1 

-1 

-1 

1 

134 

-1 

1 

1 

1 1 

-1 

1 

-1 

-1 

-1 

1 

1 

-1 

135 

-1 

1 

1 

1 1 

-1 

-1 

1 

-1 

-1 

1 

-1 

1 

136 

-1 

1 

1 

1 1 

-1 

-1 

-1 

-1 

-1 

-1 

1 

-1 

137 

-1 

1 

1 

1 -1 

1 

1 

1 

-1 

-1 

-1 

1 

-1 

138 

-1 

1 

1 

1 -1 

1 

1 

-1 

-1 

-1 

1 

-1 

1 

139 

-1 

1 

1 

1 -1 

1 

-1 

1 

-1 

-1 

1 

1 

-1 

140 

-1 

1 

1 

1 -1 

1 

-1 

-1 

-1 

-1 

-1 

-1 

1 

141 

-1 

1 

1 

1 -1 

-1 

1 

1 

-1 

1 

1 

-1 

1 

142 

-1 

1 

1 

1 -1 

-1 

1 

-1 

-1 

1 

-1 

1 

-1 

143 

-1 

1 

1 

1 -1 

-1 

-1 

1 

-1 

1 

-1 

-1 

1 
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144 -1 

145 -1 

146 -1 

147 -1 

148 -1 

149 

150 -1 

151 

152 -1 

153 -1 

154 -1 

155 

156 -1 

157 -1 

158 -1 

159 

160 

161 -1 
162 -1 

163 -1 

164 -1 

165 -1 

166 -1 

167 -1 

168 -1 

169 -1 

170 -1 

171 -1 

172 -1 

173 -1 

174 -1 

175 

176 -1 

177 -1 

178 -1 

179 -1 

180 

181 -1 
182 -1 

183 -1 

184 -1 

185 -1 

186 -1 

187 -1 

188 -1 

189 -1 

190 -1 

191 -1 

192 -1 

193 -1 


1 1 1 - 1-1 

11-111 
11-111 
11-111 
11-111 
1 1-1 1-1 

1 1-1 1-1 

1 1-1 1-1 

1 1-1 1-1 

1 1 - 1-1 1 
1 1 - 1-1 1 
1 1 - 1-1 1 
1 1 - 1-1 1 

1 1 - 1-1 -1 

1 1 - 1-1 -1 

1 1 - 1-1 -1 

1 1 - 1-1 -1 

1-1111 
1-1111 
1-11 1 1 

1-1111 
1-1 1 1-1 
1-1 1 1-1 
1-1 1 1-1 
1-1 1 1-1 
1-1 1-1 1 

1-1 1-1 1 

1-1 1-1 1 

1-1 1-1 1 

1-1 1-1 -1 

1-1 1-1 -1 

1-1 1-1 -1 

1-1 1-1 -1 

1 - 1-1 1 1 

1 - 1-1 1 1 

1 - 1-1 1 1 

1 - 1-1 1 1 

1 - 1-1 1 -1 

1 - 1-1 1 -1 

1 - 1-1 1 -1 

1 - 1-1 1 -1 

1 -1 -1 -1 1 

1 -1 -1 -1 1 

1 -1 -1 -1 1 

1 -1 -1 -1 1 

1 -1 -1 -1 -1 

1 -1 -1 -1 -1 

1 -1 -1 -1 -1 

1 -1 -1 -1 -1 

-11111 


- 1 - 1-1 1 1 1 -1 
1 1 1-1 1 - 1-1 
1-1 1 - 1-1 1 1 
-1 1 1-1 -1 -1 -1 
- 1 - 11-1111 
1111-111 
1-1 1 1 1 - 1-1 
-1111111 
- 1-111 -1 -1 -1 

1 1 1 1 -1 -1 -1 
1-111111 
-11111-1-1 
- 1 - 111-111 
111-1111 
1-1 1 - 1-1 -1 -1 

- 111 - 1-111 
-1 -1 1-1 1 -1 -1 
111-1111 
1-1 1-1 -1 -1 -1 
- 111 - 1-111 
-1 - 11-11 -1 -1 

1 1 1 1 -1 -1 -1 
1-111111 
- 11111-1 -1 
- 1 - 111-111 
1111-111 
1-1 1 1 1 - 1-1 
-1111111 
- 1-111 -1 -1 -1 

1 1 1-1 1 - 1-1 
1-1 1 - 1-1 1 1 

- 111-1 -1 -1 -1 

- 1 - 11-1111 
11 - 111-11 
1 - 1-1 1-1 1 -1 

- 11-11 -1 -1 1 

- 1 - 1-1 1 1 1 -1 

1 1 -1 -1 -1 1 -1 

1 -1 -1 - 11-11 
-1 1 - 1-1 1 1 -1 

-1 -1 -1 -1 -1 -1 1 

1 1 - 1-1 -1 -1 1 

1 -1 -1 -1 1 1 -1 

-1 1 - 1-1 1-1 1 
-1 -1 -1 -1 -1 1 -1 

11 - 1111-1 
1 - 1-1 1 -1 -1 1 

-1 1-1 1-1 1 -1 

-1 - 1-1 1 1-1 1 

11111-1-1 
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194 - 1-11111 

195 - 1-11111 

196 - 1-11111 

197 - 1-1 1 1 1-1 

198 - 1-1 1 1 1-1 

199 - 1-1 1 1 1-1 

200 -1 -1 1 1 1 -1 

201 - 1-1 1 1-1 1 

202 -1 -1 1 1 -1 1 

203 -1 -1 1 1 -1 1 

204 -1 -1 1 1 -1 1 

205 -1 -1 1 1 -1 -1 

206 -1 -1 1 1 -1 -1 

207 -1 -1 1 1 -1 -1 

208 -1 -1 1 1 -1 -1 

209 -1 -1 1 -1 1 1 

210 - 1-1 1-1 1 1 

211 - 1-1 1-1 1 1 

212 - 1-1 1-1 1 1 

213 -1 - 11-11 -1 

214 -1 - 11-11 -1 

215 -1 - 11-11 -1 

216 -1 - 11-11 -1 

217 -1 1 - 1-1 1 

218 -1 -1 1 - 1-1 1 

219 -1 -1 1 - 1-1 1 

220 -1 -1 1 -1 -1 1 

221 -1 -1 1 -1 -1 -1 

222 -1 -1 1 -1 -1 -1 

223 -1 -1 1 -1 -1 -1 

224 -1 -1 1 -1 -1 -1 

225 -1 -1 -1 1 1 1 

226 -1 -1 -1 1 1 1 

227 -1 -1 -1 1 1 1 

228 -1 -1 -1 1 1 1 

229 -1 -1 -1 1 1 -1 

230 -1 -1 -1 1 1 -1 

231 -1 - 1-1 1 1 -1 

232 -1 -1 -1 1 1 -1 

233 -1 -1 -1 1 -1 1 

234 -1 -1 -1 1 -1 1 

235 -1 -1 -1 1 -1 1 

236 -1 -1 -1 1 -1 1 

237 -1 -1 -1 1 -1 -1 

238 -1 -1 -1 1 -1 -1 

239 -1 -1 -1 1 -1 -1 

240 -1 -1 -1 1 -1 -1 

241 -1 -1 - 1-111 

242 -1 -1 -1 -1 1 1 

243 -1 -1 -1 -1 1 1 


1 - 111-111 
1 1 1 1 - 1-1 -1 

1-111111 
111 - 1-111 
1-1 1-1 1-1 -1 
111-1111 
1 -1 1 -1 -1 -1 -1 

1 1 1 -1 -1 -1 -1 

1 - 11-1111 
•1 1 1-1 1-1 -1 

■1 -1 1 - 1-1 1 1 

1111111 
1-1 1 1 - 1-1 -1 

■ 1111-111 
■ 1-1 1 1 1-1 -1 

1 1 - 1-1 1 1-1 

1 -1 -1 -1 -1 -1 1 

■1 1 -1 -1 -1 1 -1 

•1 -1 -1 -1 1-1 1 

1 1-1 1 - 1-1 1 

1 - 1-1 1 1 1-1 

• 11 - 111-11 
■1 - 1-1 1-1 1 -1 

1 1-1 1-1 1-1 

1 - 1-1 1 1-1 1 

- 11 - 1111-1 
•1 -1 -1 1 - 1-1 1 

1 1 - 1-1 1-1 1 

1 -1 -1 -1 -1 1 -1 

■1 1 -1 -1 -1 -1 1 

-1 -1 - 1-111 -1 

1 1 - 1-1 1-1 1 

1 -1 -1 -1 -1 1 -1 

•1 1 -1 -1 -1 -1 1 

•1 -1 - 1-111 -1 

1 1-1 1-1 1-1 

1 - 1-1 1 1-1 1 

•1 1-1 1 1 1-1 
■1 -1 -1 1 - 1-1 1 

1 1-1 1 - 1-1 1 

1 - 1-1 1 1 1-1 

• 11 - 111-11 
•1 -1 - 11-11 -1 

1 1 - 1-1 1 1-1 
1 -1 -1 -1 -1 -1 1 

■1 1 -1 -1 -1 1 -1 

•1 -1 -1 - 11-11 

1111111 
1-1 1 1 - 1-1 -1 
• 1111-111 
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251 
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264 
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266 

267 

268 
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270 

271 
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276 

277 

278 

279 

280 
281 
282 
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286 
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289 

290 

291 

292 

293 
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0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 


-1 

-1 

-1 

-1 

-1 

-1 

-1 

-1 

-1 

-1 

-1 

-1 

-1 

0 

0 

0 

0 

0 

0 

1 

-1 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 


1 1 - 1-1 
1-111 
1-1 1-1 
1 - 1-1 1 
1 -1 -1 -1 

-1111 
- 111-1 
- 11-11 
-11 -1 -1 
- 1-111 
-1 -1 1 -1 

-1 -1 -1 1 

-1 -1 -1 -1 

0 0 0 0 
0 0 0 0 

0 0 0 0 

0 0 0 0 

0 0 0 0 

0 0 0 0 

0 0 0 0 

0 0 0 0 

10 0 0 

-10 0 0 

0 10 0 

0-100 
0 0 10 

0 0-10 

0 0 0 1 

0 0 0 -1 

0 0 0 0 

0 0 0 0 

0 0 0 0 

0 0 0 0 

0 0 0 0 

0 0 0 0 

0 0 0 0 

0 0 0 0 

0 0 0 0 

0 0 0 0 

0 0 0 0 

0 0 0 0 

0 0 0 0 

0 0 0 0 

0 0 0 0 

0 0 0 0 

0 0 0 0 

0 0 0 0 

0 0 0 0 

0 0 0 0 

0 0 0 0 


1 1 1 - 1-1 

1 -1 -1 -1 -1 

1-1 11 1 

1-1 1-1 -1 

1 - 1-1 1 1 

1 - 1-1 1 1 

1-1 1-1 -1 

1-1111 
1 -1 -1 -1 -1 

1 1 1 - 1-1 

11-111 
1 1 - 1-1 -1 

11111 
0 0 0 0 0 

0 0 0 0 0 

0 0 0 0 0 

0 0 0 0 0 

0 0 0 0 0 

0 0 0 0 0 

0 0 0 0 0 

0 0 0 0 0 

0 0 0 0 0 

0 0 0 0 0 

0 0 0 0 0 

0 0 0 0 0 

0 0 0 0 0 

0 0 0 0 0 

0 0 0 0 0 

0 0 0 0 0 

1 0 0 0 0 

-1 0 0 0 0 

0 10 0 0 

0-1 0 0 0 
0 0 10 0 

0 0-1 0 0 

0 0 0 1 0 

0 0 0 -1 0 
0 0 0 0 1 

0 0 0 0 -1 

0 0 0 0 0 

0 0 0 0 0 

0 0 0 0 0 

0 0 0 0 0 

0 0 0 0 0 

0 0 0 0 0 

0 0 0 0 0 

0 0 0 0 0 

0 0 0 0 0 

0 0 0 0 0 

0 0 0 0 0 
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294 0 0 0 0 0 0 

295 0 0 0 0 0 0 

296 0 0 0 0 0 0 

297 0 0 0 0 0 0 

298 0 0 0 0 0 0 

299 0 0 0 0 0 0 

300 0 0 0 0 0 0 

301 0 0 0 0 0 0 

302 0 0 0 0 0 0 

303 0 0 0 0 0 0 

304 0 0 0 0 0 0 

305 0 0 0 0 0 0 

306 0 0 0 0 0 0 

307 0 0 0 0 0 0 

308 0 0 0 0 0 0 

309 0 0 0 0 0 0 

310 0 0 0 0 0 0 

311 0 0 0 0 0 0 

312 0 0 0 0 0 0 

313 0 0 0 0 0 0 

314 0 0 0 0 0 0 

315 0 0 0 0 0 0 

316 0 0 0 0 0 0 

317 0 0 0 0 0 0 

318 0 0 0 0 0 0 

319 0 0 0 0 0 0 

320 0 0 0 0 0 0 

321 0 0 0 0 0 0 

322 0 0 0 0 0 0 

323 0 0 0 0 0 0 

324 0 0 0 0 0 0 


0 0 0 0 0 0 0 
0 0 0 0 0 0 0 
0 0 0 0 0 0 0 
0 0 0 0 0 0 0 
0 0 0 0 0 0 0 
0 0 0 0 0 0 0 
0 0 0 0 0 0 0 
0 0 0 0 0 0 0 
0 0 0 0 0 0 0 
0 0 0 0 0 0 0 
0 0 0 0 0 0 0 
0 0 0 0 0 0 0 
0 0 0 0 0 0 0 
0 0 0 0 0 0 0 
0 0 0 0 0 0 0 
0 0 0 0 0 0 0 
0 0 0 0 0 0 0 
0 0 0 0 0 0 0 
0 0 0 0 0 0 0 
0 0 0 0 0 0 0 
0 0 0 0 0 0 0 
0 0 0 0 0 0 0 
0 0 0 0 0 0 0 
0 0 0 0 0 0 0 
0 0 0 0 0 0 0 
0 0 0 0 0 0 0 
0 0 0 0 0 0 0 
0 0 0 0 0 0 0 
0 0 0 0 0 0 0 
0 0 0 0 0 0 0 
0 0 0 0 0 0 0 
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APPENDIX O: 20-Term Minimal Bias Design Matrix. 


RUN X 4 XII X 23 X 24 X 25 X 31 X 32 X 33 X 34 X 36 X 39 X 43 X 62 


1111111 
2 111111 

3 111111 

4 111111 

5 11111-1 

6 11111-1 

7 11111-1 

8 11111-1 

9 1111-11 

10 1 1 1 1 -1 1 

11 1 1 1 1 -1 1 

12 1 1 1 1 -1 1 

13 1 1 1 1 - 1-1 

14 1 11 1 - 1-1 

15 1 1 1 1 - 1-1 

16 1 1 11-1 -1 

17 1 1 1 -1 1 1 

18 1 1 1 -1 1 1 

19 1 1 1 -1 1 1 

20 1 1 1 -1 1 1 

21 1 1 1 -1 1 -1 

22 1 1 1-1 1-1 

23 1 1 1-1 1-1 

24 1 1 1-1 1-1 

25 1 1 1 -1 -1 1 

26 1 1 1 - 1-1 1 

27 1 1 1 - 1-1 1 

28 1 1 1 - 1-1 1 

29 1 1 1 - 1-1 -1 

30 1 1 1 - 1-1 -1 

31 1 1 1 - 1-1 -1 

32 1 1 1 - 1-1 -1 

33 1 1 -1 1 1 1 

34 1 1 -1 1 1 1 

35 1 1 -1 1 1 1 

36 1 1 -1 1 1 1 

37 1 1-1 1 1-1 

38 1 1-1 1 1-1 

39 1 1-1 1 1-1 

40 1 1 -1 1 1 -1 

41 1 1 -1 1 -1 1 

42 1 1-1 1-1 1 

43 1 1-1 1-1 1 


1111111 

1-1 1 1 - 1-1 -1 

-1111-111 
- 1 - 1111-1 -1 
1 1 1 -1 -1 -1 -1 

1 - 11-1111 
-1 1 1-1 1-1 -1 

-1 -1 1 - 1-1 1 1 

111 - 1-111 
1-1 1-1 1-1 -1 

- 111-1111 
-1 -1 1 -1 -1 -1 -1 

11111 - 1-1 
1 - 111-111 
-1 1 1 1 - 1-1 -1 

- 1-111111 
1 1 - 1-1 1-1 1 

1 -1 -1 -1 -1 1 -1 

-1 1 -1 -1 -1 -1 1 

-1 -1 - 1-111 -1 
1 1-1 1-1 1-1 
1 - 1-1 1 1-1 1 

- 11 - 1111-1 
-1 -1 -1 1 - 1-1 1 

1 1-1 1 - 1-1 1 

1 - 1-1 1 1 1-1 

- 11 - 111-11 
-1 -1 -1 1-1 1 -1 
1 1 - 1-1 1 1-1 

1 -1 -1 -1 -1 -1 1 

-1 1 -1 -1 -1 1 -1 

-1 -1 -1 - 11-11 

1 1 - 1-1 1 1-1 

1 -1 -1 -1 -1 -1 1 

-1 1 -1 -1 -1 1 -1 

-1 -1 -1 - 11-11 

1 1-1 1 - 1-1 1 

1 - 1-1 1 1 1-1 

- 11 - 111-11 
-1 -1 - 11-11 -1 

1 1-1 1-1 1-1 

1 - 1-1 1 1-1 1 

- 11 - 1111-1 
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44 

45 

46 

47 

48 

49 

50 

51 

52 

53 

54 

55 

56 

57 

58 

59 

60 
61 
62 

63 

64 

65 

66 

67 

68 

69 

70 

71 

72 

73 

74 

75 

76 

77 

78 

79 

80 
81 
82 

83 

84 

85 

86 

87 

88 

89 

90 

91 

92 

93 


1 1 -1 

1 1 -1 

1 1 -1 

1 1 -1 

1 1 -1 

1 1 -1 

1 1 -1 

1 1 -1 

1 1 -1 

1 1 -1 

1 1 -1 

1 1 -1 

1 1 -1 

1 1 -1 

1 1 -1 

1 1 -1 

1 1 -1 

1 1 -1 

1 1 -1 

1 1 -1 

1 1 -1 

1 -1 1 

1 -1 1 

1 -1 1 

1 -1 1 

1-1 1 
1 -1 1 

1 -1 1 

1 -1 1 

1 -1 1 

1 -1 1 

1 -1 1 

1 -1 1 

1 -1 1 

1 -1 1 

1 -1 1 

1 -1 1 

1 -1 1 

1 -1 1 

1 -1 1 

1 -1 1 

1 -1 1 

1 -1 1 

1 -1 1 

1 -1 1 

1 -1 1 

1 -1 1 

1 -1 1 

1 -1 1 

1 -1 1 


1-1 1-1 -1 -1 1 

1 - 1-1 1 1-1 -1 
1 - 1-1 1 -1 -1 -1 

1 -1 -1 -1 1 -1 -1 

1 -1 -1 -1 -1 -1 -1 

-1111111 
- 1111-111 
- 111-1111 
- 111 - 1-111 
- 11 - 1111-1 
-1 1-1 1-1 1 -1 

- 11 - 1-111 -1 
-1 1 -1 -1 -1 1 -1 
- 1 - 11111-1 
- 1-1 1 1-1 1 -1 

-1 - 11-11 1 -1 
-1 -1 1 - 1-1 1 -1 

- 1 - 1-11111 
-1 - 1-1 1-1 1 1 

-1 - 1 - 1-1 1 1 1 

-1 -1 -1 - 1-111 
11111-11 
1111 - 1-11 
111 - 11-11 
1 1 1 -1 -1 -1 1 
1 1-1 1 1 - 1-1 

1 1-1 1 - 1-1 -1 

1 1 - 1-1 1-1 -1 

1 1 - 1-1 -1 -1 -1 

1-1 1 1 1 - 1-1 

1-1 1 1 - 1-1 -1 

1-1 1-1 1 - 1-1 

1-1 1-1 -1 -1 -1 

1 - 1-1 1 1-1 1 

1-1-1 1 -1 -1 1 

1 -1 -1 - 11-11 
1 -1 -1 -1 -1 -1 1 

- 111111-1 
-1111-11-1 
- 111 - 111-1 
-1 1 1-1-1 1 -1 

- 11-11111 
- 11 - 11-111 
- 11 - 1-1111 
-1 1 -1 -1 -1 1 1 

- 1-111111 
- 1 - 111-111 
- 1 - 11-1111 
-1 -1 1 - 1-1 1 1 

-1-1-1 1 1 1 -1 


-1 -1 

1 -1 

-1 1 

-1 -1 

1 1 

1 -1 

-1 1 

-1 -1 

1 1 

-1 1 

1 -1 

1 1 

-1 -1 

-1 -1 

1 1 

1 -1 

-1 1 

1 1 

-1 -1 

-1 1 

1 -1 

1 -1 

-1 1 

-1 -1 

1 1 

-1 1 

1 -1 

1 1 

-1 -1 

-1 -1 

1 1 

1 -1 

-1 1 

1 1 

-1 -1 

-1 1 

1 -1 

1 1 

-1 -1 

-1 1 

1 -1 

-1 -1 

1 1 

1 -1 

-1 1 

-1 1 

1 -1 

1 1 

-1 -1 

1 -1 


1 

1 

-1 

1 

-1 

-1 

1 

-1 

1 

1 

-1 

1 

-1 

-1 

1 

-1 

1 

1 

-1 

1 

-1 

1 

-1 

1 

-1 

-1 

1 

-1 

1 

1 

-1 

1 

-1 

-1 

1 

-1 

1 

1 

-1 

1 

-1 

-1 

1 

-1 

1 

1 

-1 

1 

-1 

-1 
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94 

1 

-1 

1 

95 

1 

-1 

1 

96 

1 

-1 

1 

97 

1 

-1 

-1 

98 

1 

-1 

-1 

99 

1 

-1 

-1 

100 

1 

-1 

-1 

101 

1 

-1 

-1 

102 

1 

-1 

-1 

103 

1 

-1 

-1 

104 

1 

-1 

-1 

105 

1 

-1 

-1 

106 

1 

-1 

-1 

107 

1 

-1 

-1 

108 

1 

-1 

-1 

109 

1 

-1 

-1 

110 

1 

-1 

-1 

111 

1 

-1 

-1 

112 

1 

-1 

-1 

113 

1 

-1 

-1 

114 

1 

-1 

-1 

115 

1 

-1 

-1 

116 

1 

-1 

-1 

117 

1 

-1 

-1 

118 

1 

-1 

-1 

119 

1 

-1 

-1 

120 

1 

-1 

-1 

121 

1 

-1 

-1 

122 

1 

-1 

-1 

123 

1 

-1 

-1 

124 

1 

-1 

-1 

125 

1 

-1 

-1 

126 

1 

-1 

-1 

127 

1 

-1 

-1 

128 

1 

-1 

-1 

129 

-1 

1 

1 

130 

-1 

1 

1 

131 

-1 

1 

1 

132 

-1 

1 

1 

133 

-1 

1 

1 

134 

-1 

1 

1 

135 

-1 

1 

1 

136 

-1 

1 

1 

137 

-1 

1 

1 

138 

-1 

1 

1 

139 

-1 

1 

1 

140 

-1 

1 

1 

141 

-1 

1 

1 

142 

-1 

1 

1 

143 

-1 

1 

1 


-1 -1 

-1 -1 

-1 -1 

1 1 

1 1 

1 1 

1 1 

1 1 

1 1 

1 1 

1 1 

1 -1 

1 -1 

1 -1 

1 -1 

1 -1 

1 -1 

1 -1 

1 -1 

-1 1 

-1 1 

-1 1 

-1 1 

-1 1 

-1 1 

-1 1 

-1 1 

-1 -1 

-1 -1 

-1 -1 

-1 -1 

-1 -1 

-1 -1 

-1 -1 

-1 -1 

1 1 

1 1 

1 1 

1 1 

1 1 

1 1 

1 1 

1 1 

1 -1 

1 -1 

1 -1 

1 -1 

1 -1 

1 -1 

1 -1 


- 1-111 
-1 -1 -1 1 

1111 
11-11 

1-111 
1 - 1-1 1 

-1111 
- 11-11 
-1-111 
-1 -1 -1 1 

1111 

11-11 

1-111 

1 - 1-1 1 

-1111 
- 11-11 
- 1-1 1 1 
-1 -1 -1 1 

111-1 
1 1 - 1-1 
1-1 1-1 
1 -1 -1 -1 

- 111-1 
-1 1 -1 -1 

-1 -1 1 -1 

-1 -1 -1 -1 

111-1 
1 1 - 1-1 
1-1 1-1 
1 -1 -1 -1 

- 111-1 
-1 1 -1 -1 

-1 -1 1 -1 

-1 -1 -1 -1 

111-1 
1 1 - 1-1 
1-1 1-1 
1 -1 -1 -1 

- 111-1 
-1 1 -1 -1 

-1 -1 1 -1 

-1 -1 -1 -1 

111-1 
1 1 - 1-1 
1-1 1-1 
1 -1 -1 -1 

- 111-1 
-1 1 -1 -1 

-1 -1 1 -1 


- 1-111 
-1 -1 -1 -1 

-1111 
-1 1 -1 -1 

- 1-1 1 1 
-1 -1 -1 -1 

-1111 
1-111 
1 1 - 1-1 
1111 
1 -1 -1 -1 

1 -1 -1 -1 

1111 
1 1 - 1-1 
1-111 
-1111 
-1 -1 -1 -1 

-1-111 
-1 1 -1 -1 

111-1 
1 - 1-1 1 
1-1 1-1 

11-11 
-1 -1 -1 1 

- 111-1 
- 11-11 
-1 -1 1 -1 

-1 -1 1 -1 

-1 1-1 1 

- 111-1 
-1 -1 -1 1 

11-11 
1-1 1-1 
1 - 1-1 1 

111-1 

111-1 
1 - 1-1 1 
1-1 1-1 
11-11 
-1 -1 -1 1 

- 111-1 
-1 1 -1 1 

-1 -1 1 -1 

-1 -1 1 -1 

- 11-11 
-1 1 1-1 

-1 -1 -1 1 

11-11 
1-1 1-1 
1 - 1-1 1 
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APPENDIX P: 20-Term MSLK Deviation Matrix. 
Axial points not included 
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245 0 0 0 0 

246 0 0 0 0 

247 0 0 0 0 

248 0 0 0 0 

249 0 0 0 0 

250 0 0 0 0 

251 0 0 0 0 

252 -1 0 0 0 

253 -1 -1 0 0 

254 -1 -1 -1 0 

255 -1 -1 -1 -1 

256 -1 -1 -1 -1 

257 -5111 

258 0 -5 1 1 

259 0 0 -5 1 

260 0 0 0 -5 

261 0 0 0 0 

262 0 0 0 0 

263 0 0 0 0 

264 0 0 0 0 

265 0 0 0 0 

266 0 0 0 0 

267 0 0 0 0 

268 1 0 0 0 

269 1 1 0 0 

270 1 1 1 0 

271 1 1 1 1 

272 1 1 1 1 

273 0 0 0 0 

274 0 0 0 0 

275 0 0 0 0 

276 0 0 0 0 

277 0 0 0 0 

278 0 0 0 0 

279 0 0 0 0 

280 0 0 0 0 

281 0 0 0 0 

282 0 0 0 0 

283 0 0 0 0 

284 0 0 0 0 


5 -1 -1 -1 -1 -1 

0 5 - 1-1 -1 -1 

0 0 5 -1 -1 -1 

0 0 0 5 -1 -1 

0 0 0 0 5 -1 

0 0 0 0 0 5 

0 0 0 0 0 0 

0 0 0 0 0 0 

oooooo 

0 0 0 0 0 0 

oooooo 

-1 0 0 0 0 0 

1 1 0 0 0 0 

1110 0 0 
11110 0 
111110 
-511111 
0-51111 
0 0-51 1 1 

0 0 0 -5 1 1 

0 0 0 0 -5 1 

0 0 0 0 0 -5 

OOOOOO 

oooooo 
oooooo 
oooooo 
oooooo 
1 0 0 0 0 0 

oooooo 
oooooo 
oooooo 
oooooo 
oooooo 
oooooo 
oooooo 
oooooo 
oooooo 
oooooo 
oooooo 
oooooo 


oooooo 

-1 0 0 0 0 0 

-1 -1 0 0 0 0 

- 1 - 1-1 0 0 0 

-1 -1 - 1-1 0 0 
-1 -1 -1 -1 -1 o 

5 -1 -1 -1 -1 -1 

0 5 - 1-1 -1 -1 

0 0 5 -1 -1 -1 

0 0 0 5 -1 -1 

0 0 0 0 5 -1 

0 0 0 0 0 5 

oooooo 
oooooo 
oooooo 
oooooo 
oooooo 
1 0 0 0 0 0 

1 1 0 0 0 0 

1110 0 0 
11110 0 
111110 
-511111 
0-51111 
0 0-51 1 1 

0 0 0 -5 1 1 

0 0 0 0 -5 1 

0 0 0 0 0 -5 

oooooo 
oooooo 
oooooo 
oooooo 
oooooo 
oooooo 
oooooo 
oooooo 
oooooo 
oooooo 
oooooo 
oooooo 
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APPENDIX Q: SSN CCD Design Matrix. 
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97 1 -1 -1 

98 1 -1 -1 

99 1 -1 -1 

100 1 -1 -1 

101 1 -1 -1 

102 1 -1 -1 

103 1 -1 -1 

104 1 -1 -1 

105 1 -1 -1 

106 1 -1 -1 

107 1 -1 -1 

108 1 -1 -1 

109 1 -1 -1 

110 1 -1 -1 

111 1 -1 -1 

112 1 -1 -1 

113 1 -1 -1 

114 1 -1 -1 

115 1 -1 -1 

116 1 -1 -1 

117 1 -1 -1 

118 1 -1 -1 

119 1 -1 -1 

120 -1 -1 

121 -1 -1 

122 -1 -1 

123 1 -1 -1 

124 1 -1 -1 

125 1 -1 -1 

126 1 -1 -1 

127 1 -1 -1 

128 -1 -1 

129 -1 1 1 

130 -1 1 1 

131 -1 1 1 

132 -1 1 1 

133 -1 1 1 

134 -1 1 1 

135 -1 1 1 

136 -1 1 1 

137 -1 1 1 

138 -11 1 

139 -1 1 1 

140 -1 1 1 

141 -1 1 1 

142 -1 1 1 

143 -1 1 1 

144 -1 1 1 

145 -1 1 1 

146 -1 1 1 


11111 
1111-1 
111-11 
1 1 1 - 1-1 
11-111 
1 1-1 1-1 

1 1 - 1-1 1 

1 1 - 1-1 -1 

1-1111 
1-1 1 1-1 

1-1 1-1 1 

1-1 1-1 -1 

1 - 1-1 1 1 

1 - 1-1 1 -1 

1 -1 -1 -1 1 

1 -1 -1 -1 -1 

-11111 
- 1111-1 
- 111-11 
- 111-1 -1 
- 11-111 
- 11-11 -1 
-1 1 - 1-1 1 
-1 1 -1 -1 -1 

- 1-1111 
- 1-111 -1 
-1 - 11-11 
-1 -1 1 -1 -1 

-1 - 1-111 
-1 -1 -1 1 -1 

-1 -1 -1 -1 1 

-1 -1 -1 -1 -1 

11111 
1111-1 
111-11 
1 1 1 - 1-1 
11-111 
1 1-1 1-1 

1 1 - 1-1 1 

1 1 - 1-1 -1 

1-1111 
1-1 1 1-1 

1-1 1-1 1 

1-1 1-1 -1 

1 - 1-1 1 1 

1 - 1-1 1 -1 

1 -1 -1 -1 1 

1 -1 -1 -1 -1 

-11111 
- 1111-1 


1-1 1-1 -1 

1 - 1-1 1 1 
1 -1 -1 -1 -1 
1-1111 
11-111 
1 1 1 - 1-1 

11111 
1 1 - 1-1 -1 

1 1 - 1-1 -1 

11111 
1 1 1 - 1-1 

11-111 
1-1111 
1 -1 -1 -1 -1 

1 - 1-1 1 1 

1 -1 1 -1 -1 

- 1-111 -1 
-1 -1 -1 -1 1 

-1 -1 -1 1 -1 

-1 -1 1-1 1 

-1 1 - 1-1 1 

- 1111-1 
- 111-11 
- 11-11 -1 

- 11-11 -1 

- 111-11 
- 1111-1 
-1 1 - 1-1 1 

-1 -1 1-1 1 

-1 -1 -1 1 -1 

-1 -1 -1 -1 1 

- 1-111 -1 

- 1-111 -1 

-1 -1 -1 -1 1 

-1 -1 -1 1 -1 

-1 - 11-11 
-1 1 - 1-1 1 

- 1111-1 
- 111-11 
- 11-11 -1 

- 11-11 -1 

- 111-11 
- 1111-1 
-1 1 - 1-1 1 

-1 -1 1-1 1 
-1 -1 -1 1 -1 

-1 -1 -1 -1 1 

-1-111 -1 

1-1 1-1 -1 

1 - 1-1 1 1 
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APPENDIX R: 20-Term Design Centers. 
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APPENDIX S, SSN Design Center. 
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estimate the impact of operating at non-optimal decision variable values. This work develops and demonstrates a technique 
for characterizing the surface using response surface methodology. Specifically, the flexibility and utility of RSM techniques 
applied to this class of problems is demonstrated, and a methodology for characterizing the surface in the local region using a 
low-order polynomial is developed. 
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